source_monitor 0.7.0 → 0.8.0

data/.vbw-planning/SHIPPED.md

@@ -1,63 +0,0 @@
-# Shipped Milestones
-
-## default (2026-02-09 to 2026-02-10)
-
-**Core value:** Drop-in Rails engine for feed monitoring, content scraping, and operational dashboards.
-
-### Metrics
-
-| Metric | Value |
-|--------|-------|
-| Phases | 4 |
-| Plans completed | 14 |
-| Commits | 18 |
-| Requirements satisfied | 15/15 |
-| Test runs | 841 (up from 473) |
-| Coverage | 86.97% line (510 uncovered, down from 2117) |
-
-### Phases
-
-1. Coverage Analysis & Quick Wins (2 plans)
-2. Critical Path Test Coverage (5 plans)
-3. Large File Refactoring (4 plans)
-4. Code Quality & Conventions Cleanup (3 plans)
-
-### Archive
-
-Location: `.vbw-planning/milestones/default/`
-Tag: `milestone/default`
-
----
-
-## upgrade-assurance (2026-02-12 to 2026-02-13)
-
-**Goal:** Give host app developers confidence that gem updates go smoothly -- automated migration detection, upgrade command, config validation, and AI-assisted upgrade guidance.
-
-### Metrics
-
-| Metric | Value |
-|--------|-------|
-| Phases | 3 |
-| Plans completed | 3 |
-| Tasks completed | 14 |
-| Commits | 12 |
-| Requirements satisfied | 5/5 (REQ-26 through REQ-30) |
-| Tests | 1003 (up from 973) |
-
-### Phases
-
-1. Upgrade Command & Migration Verifier (1 plan, 5 tasks)
-2. Configuration Deprecation Framework (1 plan, 4 tasks)
-3. Upgrade Skill & Documentation (1 plan, 5 tasks)
-
-### Key Decisions
-
-- 3 phases: command, config, skill -- each independently valuable
-- `.source_monitor_version` marker file for version tracking
-- Deprecation registry with :warning and :error severities
-- sm-upgrade as a consumer skill (installed by default)
-
-### Archive
-
-Location: `.vbw-planning/milestones/upgrade-assurance/`
-Tag: `milestone/upgrade-assurance`

data/.vbw-planning/STATE.md

@@ -1,27 +0,0 @@
-# State
-
-## Current Position
-
-- **Milestone:** aia-ssl-fix
-- **Phase:** 1 -- AIA Certificate Resolution
-- **Status:** Complete
-- **Progress:** 100%
-
-## Decisions
-
-| Decision | Date | Context |
-|----------|------|---------|
-| Single-phase milestone for AIA fix | 2026-02-17 | Complete plan already validated; no scoping needed |
-| 3 plans with wave parallelism | 2026-02-17 | Plans 01+02 (wave 1, disjoint files), Plan 03 (wave 2, integration) |
-
-## Todos
-
-## Metrics
-
-- **Started:** 2026-02-17
-- **Phases:** 1
-- **Plans:** 3
-- **Tests at start:** 1003
-- **Tests at end:** 1025
-- **Commits:** 4 (f60e9bf, 4c9568a, 9c38bc3, e68a6b0)
-- **Plans completed:** 3/3

data/.vbw-planning/codebase/ARCHITECTURE.md

@@ -1,147 +0,0 @@
-# Architecture
-
-## System Overview
-
-SourceMonitor is a mountable Rails 8 engine that ingests RSS, Atom, and JSON feeds, scrapes full article content, and surfaces Solid Queue-powered dashboards for monitoring and remediation. It is packaged as a RubyGem and mounted into a host Rails application.
-
-## Architecture Pattern
-
-**Mountable Rails Engine** with `isolate_namespace SourceMonitor`. The engine:
-- Provides its own models, controllers, views, jobs, and frontend assets
-- Uses its own `ApplicationRecord`, `ApplicationController`, and `ApplicationJob` base classes
-- Namespaces all database tables under a configurable prefix (default: `sourcemon_`)
-- Registers its own routes, asset paths, and initializers
-
-## Core Domain Modules
-
-### 1. Feed Fetching (`lib/source_monitor/fetching/`)
-The primary data ingestion pipeline:
-- `FeedFetcher` -- Orchestrates HTTP request, feed parsing via Feedjira, item creation, adaptive interval scheduling, and retry policies
-- `FetchRunner` -- Entry point for enqueuing fetch jobs; handles concurrency control
-- `FetchError` hierarchy -- Typed error classes (TimeoutError, ConnectionError, HTTPError, ParsingError)
-- `RetryPolicy` -- Exponential backoff with circuit breaker pattern
-- `StalledFetchReconciler` -- Recovers stalled fetch jobs
-
-### 2. Content Scraping (`lib/source_monitor/scraping/` + `lib/source_monitor/scrapers/`)
-Pluggable content extraction system:
-- `Scrapers::Base` -- Abstract adapter contract; subclasses implement `#call` returning a `Result` struct
-- `Scrapers::Readability` -- Default adapter using ruby-readability
-- `Scraping::ItemScraper` -- Orchestrator that resolves adapter, executes scrape, persists results
-- `Scraping::BulkSourceScraper` -- Batch scraping across all items for a source
-- `Scraping::Enqueuer` -- Manages scrape job queuing with in-flight throttling
-- `Scraping::State` -- Tracks in-flight scrape state via cache/memory
-
-### 3. Health Monitoring (`lib/source_monitor/health/`)
-Source health tracking system:
-- `SourceHealthMonitor` -- Computes health status from recent fetch history (sliding window)
-- `SourceHealthCheck` -- One-off health probe for a source URL
-- `ImportSourceHealthCheck` -- Variant for import wizard health checks
-- `SourceHealthReset` -- Resets health status for a source
-- Configurable thresholds: healthy (0.8), warning (0.5), auto-pause (0.2)
-
-### 4. Scheduling (`lib/source_monitor/scheduler.rb`)
-- `Scheduler` -- Periodic job that finds sources due for fetch using `FOR UPDATE SKIP LOCKED`
-- Adaptive fetch interval algorithm (increase/decrease factors, jitter)
-- Integrates with `StalledFetchReconciler` for recovery
-
-### 5. Event System (`lib/source_monitor/events.rb`)
-- Typed event structs: `ItemCreatedEvent`, `ItemScrapedEvent`, `FetchCompletedEvent`
-- Callback-based dispatch with error isolation per handler
-- Item processor pipeline for custom host-app processing
-- Registration via `SourceMonitor.config.events.after_*` DSL
-
-### 6. Configuration (`lib/source_monitor/configuration.rb`)
-Rich nested configuration object with sub-configs:
-- `HTTPSettings` -- timeout, retries, user agent, proxy, headers
-- `ScraperRegistry` -- pluggable adapter registration
-- `RetentionSettings` -- item retention days, max items, strategy (destroy/soft_delete)
-- `RealtimeSettings` -- adapter selection (solid_cable/redis/async)
-- `FetchingSettings` -- adaptive interval tuning
-- `HealthSettings` -- health window and threshold configuration
-- `AuthenticationSettings` -- pluggable authentication/authorization handlers
-- `ScrapingSettings` -- concurrency limits
-- `Events` -- callback registration
-- `Models` -- concern injection and custom validation registration
-
-### 7. Model Extensions (`lib/source_monitor/model_extensions.rb`)
-Dynamic model customization system:
-- Host apps can inject concerns and validations into engine models at configuration time
-- `ModelExtensions.register(model_class, key)` -- called in each model class body
-- `ModelExtensions.reload!` -- re-applies all extensions (called on configuration change)
-- Manages table name assignment from configurable prefix
-
-### 8. Real-time Broadcasting (`lib/source_monitor/realtime/`)
-- `Realtime::Adapter` -- Configures Action Cable based on selected adapter
-- `Realtime::Broadcaster` -- Broadcasts source/item updates and toast notifications
-- `Dashboard::TurboBroadcaster` -- Wires dashboard stat updates to Turbo Streams
-
-### 9. Setup/Installation System (`lib/source_monitor/setup/`)
-Comprehensive host-app installation workflow:
-- `Setup::CLI` -- Command-line interface for setup
-- `Setup::Workflow` -- Orchestrates multi-step installation
-- `Setup::Requirements` / `Setup::Detectors` -- System requirement checks
-- `Setup::GemfileEditor` / `Setup::BundleInstaller` / `Setup::NodeInstaller` -- Dependency installation
-- `Setup::InstallGenerator` -- Rails generator for migrations, routes, initializer
-- `Setup::Verification::Runner` -- Post-install verification (Solid Queue, Action Cable)
-
-### 10. OPML Import Wizard (`app/controllers/source_monitor/import_sessions_controller.rb`)
-Multi-step wizard for bulk feed import:
-- 5-step flow: upload -> preview -> health_check -> configure -> confirm
-- State persisted in `ImportSession` model with JSONB columns
-- Health check jobs run asynchronously with Turbo Stream progress updates
-- Bulk settings applied to all imported sources
-
-## Data Model
-
-```
-Source (sourcemon_sources)
-  |-- has_many Item (sourcemon_items)
-  |     |-- has_one ItemContent (sourcemon_item_contents) [separate table for large content]
-  |     |-- has_many ScrapeLog (sourcemon_scrape_logs)
-  |     +-- has_many LogEntry (sourcemon_log_entries) [polymorphic]
-  |-- has_many FetchLog (sourcemon_fetch_logs)
-  |-- has_many HealthCheckLog (sourcemon_health_check_logs)
-  +-- has_many LogEntry (sourcemon_log_entries)
-
-LogEntry (sourcemon_log_entries)
-  |-- delegated_type :loggable -> FetchLog | ScrapeLog | HealthCheckLog
-
-ImportSession (sourcemon_import_sessions)
-  +-- JSONB columns for wizard state
-
-ImportHistory (sourcemon_import_histories)
-  +-- Records completed imports
-```
-
-## Job Architecture
-
-All jobs inherit from `SourceMonitor::ApplicationJob` which inherits from the host app's `ApplicationJob` (or `ActiveJob::Base`).
-
-| Job | Queue | Purpose |
-|-----|-------|---------|
-| `ScheduleFetchesJob` | fetch | Recurring: triggers Scheduler to find and enqueue due sources |
-| `FetchFeedJob` | fetch | Fetches a single source's feed, creates items |
-| `ScrapeItemJob` | scrape | Scrapes content for a single item |
-| `SourceHealthCheckJob` | fetch | Runs health check for a source |
-| `ImportSessionHealthCheckJob` | fetch | Health checks during OPML import wizard |
-| `ImportOpmlJob` | fetch | Bulk-creates sources from OPML import |
-| `LogCleanupJob` | fetch | Recurring: prunes old log entries |
-| `ItemCleanupJob` | fetch | Recurring: prunes old items per retention policy |
-
-Queue names are configurable via `config.fetch_queue_name` and `config.scrape_queue_name`.
-
-## Security Architecture
-
-- `Security::Authentication` -- Pluggable authentication via handler callbacks (symbol method names or callables)
-- `Security::ParameterSanitizer` -- HTML sanitization of all user inputs via `ActionView::Base.full_sanitizer`
-- `Models::Sanitizable` -- Concern that sanitizes string and hash model attributes before validation
-- `Models::UrlNormalizable` -- URL normalization and validation concern
-- `SanitizesSearchParams` -- Controller concern for search parameter sanitization
-- CSRF protection enabled (`protect_from_forgery with: :exception`)
-- No built-in user model -- delegates to host app
-
-## Instrumentation & Metrics
-
-- `Instrumentation` -- Emits `ActiveSupport::Notifications` events for fetch lifecycle
-- `Metrics` -- In-memory counters and gauges, populated via notification subscribers
-- Events: `source_monitor.fetch.start`, `source_monitor.fetch.finish`, `source_monitor.scheduler.run`, `source_monitor.items.duplicate`, `source_monitor.items.retention`

data/.vbw-planning/codebase/CONCERNS.md

@@ -1,99 +0,0 @@
-# Concerns
-
-## Technical Debt
-
-### 1. Large Files
-- `lib/source_monitor/fetching/feed_fetcher.rb` (627 lines) -- The core fetch pipeline handles HTTP, parsing, item creation, adaptive intervals, retry strategies, and source updates all in one class. This is the largest single-responsibility violation.
-- `lib/source_monitor/configuration.rb` (655 lines) -- Contains ~12 nested configuration classes in a single file. Each settings class could be extracted.
-- `app/controllers/source_monitor/import_sessions_controller.rb` (792 lines) -- The OPML import wizard controller handles all 5 wizard steps, file parsing, health checks, selection management, and bulk operations.
-- `lib/source_monitor/items/item_creator.rb` -- Large item creation file (based on coverage baseline entry count).
-- `config/coverage_baseline.json` (2329 lines) -- This file itself is very large, indicating significant uncovered code.
-
-### 2. Coverage Baseline Gaps
-The `config/coverage_baseline.json` catalogs uncovered lines. Particularly notable gaps:
-- `lib/source_monitor/items/item_creator.rb` -- Hundreds of uncovered branch lines
-- `lib/source_monitor/fetching/feed_fetcher.rb` -- Extensive uncovered branches
-- `lib/source_monitor/configuration.rb` -- Many configuration edge cases untested
-- `lib/source_monitor/dashboard/queries.rb` -- Dashboard query logic largely uncovered
-- `lib/source_monitor/realtime/broadcaster.rb` -- Broadcasting logic has gaps
-- `lib/source_monitor/scraping/bulk_source_scraper.rb` -- Bulk scraping coverage gaps
-- `lib/source_monitor/analytics/sources_index_metrics.rb` -- Analytics largely uncovered
-
-### 3. LogEntry Hard-coded Table Name
-In `app/models/source_monitor/log_entry.rb` line 6: `self.table_name = "sourcemon_log_entries"` -- This bypasses the configurable table name prefix system that all other models use via `ModelExtensions.register`. The table name is hard-coded despite the engine supporting custom prefixes.
-
-### 4. `lib/source_monitor.rb` Entry Point Complexity
-The main entry point has 102+ require statements loaded eagerly. This means all engine code is loaded at boot regardless of what features the host app uses. No autoloading or lazy-loading strategy.
-
-## Architectural Risks
-
-### 1. PostgreSQL Lock-in
-The `Scheduler` uses `FOR UPDATE SKIP LOCKED` (PostgreSQL-specific locking), and queries use `NULLS FIRST`/`NULLS LAST` ordering. The engine will not work with MySQL or SQLite. This is documented nowhere in the gemspec constraints.
-
-### 2. In-Memory Metrics Without Persistence
-`SourceMonitor::Metrics` stores counters and gauges in module-level instance variables (`@counters`, `@gauges`). These are:
-- Lost on process restart
-- Not shared across workers/processes
-- Not suitable for production monitoring
-
-### 3. Scraping State Management
-`SourceMonitor::Scraping::State` tracks in-flight scrapes but the mechanism (cache/memory) is fragile. In multi-process deployments, this could lead to over-enqueuing or under-enqueuing of scrape jobs.
-
-### 4. Optional Dependency Loading
-Solid Queue, Solid Cable, Turbo, and Ransack are loaded with `rescue LoadError`. This means the engine could silently fail to load core functionality without clear error messages to the host app developer.
-
-### 5. No Database Index Verification
-24 migration files create the schema incrementally. There is no single-schema check to verify all expected indexes exist after migration.
-
-## Security Considerations
-
-### 1. No Built-in Authentication
-The engine relies entirely on the host app for authentication. The `Security::Authentication` module provides hooks but no default protection. An unconfigured engine is accessible to anyone who can reach the mount path.
-
-### 2. Fallback User Creation in Import Controller
-`ImportSessionsController#create_guest_user` (lines 416-429) can create a `User` record in the host app's database when no authentication is configured. This is a defensive fallback but could be unexpected behavior.
-
-### 3. Parameter Sanitization Scope
-`Security::ParameterSanitizer` uses `ActionView::Base.full_sanitizer` which strips all HTML. This is broad but may not catch all injection vectors (e.g., URL-based attacks, header injection via custom_headers).
-
-### 4. Custom Headers Passthrough
-Source `custom_headers` are stored as JSONB and passed to HTTP requests (`FeedFetcher#request_headers`). Malicious header values could potentially be used for SSRF or header injection if not properly validated.
-
-## Performance Considerations
-
-### 1. N+1 Query Risk in Dashboard
-`Dashboard::Queries` performs multiple database queries for stats, recent activity, quick actions, job metrics, and fetch schedules. The controller assembles all of these synchronously.
-
-### 2. Large OPML File Handling
-The OPML import parses entire files in memory (`Nokogiri::XML(content)`). Large OPML files with thousands of entries could cause memory pressure.
-
-### 3. Bulk Scrape Operations
-`BulkSourceScraper` enqueues individual `ScrapeItemJob` for each item. For sources with thousands of items, this could flood the job queue.
-
-### 4. Coverage Baseline File
-The `config/coverage_baseline.json` at 2329 lines is parsed at test time and represents significant technical debt in test coverage.
-
-## Operational Risks
-
-### 1. No Health Check Endpoint Documentation
-The `/health` endpoint exists but its response format and behavior are not documented.
-
-### 2. Stalled Fetch Recovery
-The `StalledFetchReconciler` runs within the `Scheduler`, but if the scheduler itself stalls, there is no external recovery mechanism.
-
-### 3. Circuit Breaker State in Database
-Fetch circuit breaker state (`fetch_circuit_opened_at`, `fetch_circuit_until`) is stored on the `Source` model. This means circuit breaker resets require database writes, and a failing database connection prevents circuit recovery.
-
-## Dependency Risks
-
-### 1. ruby-readability (~> 0.7)
-This gem appears to be minimally maintained. The `0.7.x` line is old. It depends on Nokogiri which receives frequent security updates, creating a transitive dependency management burden.
-
-### 2. Nokolexbor (~> 0.5)
-Native extension gem with C bindings to the Lexbor HTML parser. Platform-specific builds could cause deployment issues, especially on less common architectures.
-
-### 3. Feedjira Wide Version Range
-`>= 3.2, < 5.0` allows major version bumps that could introduce breaking API changes.
-
-### 4. Ruby >= 3.4.0 Minimum
-This is an aggressive minimum that excludes many production Ruby installations still on 3.2.x or 3.3.x.

data/.vbw-planning/codebase/CONVENTIONS.md

@@ -1,97 +0,0 @@
-# Conventions
-
-## Ruby Code Style
-
-- **Style Guide**: Rails Omakase via `rubocop-rails-omakase` gem
-- **Frozen String Literals**: Consistently used across all app/ and lib/ Ruby files (`# frozen_string_literal: true`)
-- **No frozen_string_literal**: Absent from some test files and the main `lib/source_monitor.rb` entry point
-
-## Naming Conventions
-
-### Module/Class Naming
-- All engine code namespaced under `SourceMonitor::`
-- Sub-modules use descriptive domain names: `Fetching`, `Scraping`, `Health`, `Security`, `Dashboard`, `Realtime`
-- Service objects named as nouns (e.g., `FeedFetcher`, `ItemScraper`, `Scheduler`, `RetentionPruner`)
-- Presenters suffixed with `Presenter` (e.g., `BulkResultPresenter`, `RecentActivityPresenter`, `TurboStreamPresenter`)
-- Query objects in `analytics/` and `dashboard/` (e.g., `Queries`, `SourcesIndexMetrics`)
-
-### File Organization
-- One class per file, file path mirrors class namespace
-- Nested classes extracted to sub-directories (e.g., `item_scraper/adapter_resolver.rb`, `item_scraper/persistence.rb`)
-- Concerns placed in `concerns/` subdirectories
-
-### Database Naming
-- Table prefix: `sourcemon_` (configurable via `config.models.table_name_prefix`)
-- Tables: `sourcemon_sources`, `sourcemon_items`, `sourcemon_fetch_logs`, etc.
-- Foreign keys follow Rails conventions: `source_id`, `item_id`
-- Timestamps use `started_at`/`completed_at` pattern for log records
-
-### Method Naming
-- Bang methods (`!`) for operations that raise on failure: `soft_delete!`, `mark_processing!`
-- Predicate methods: `fetch_circuit_open?`, `auto_paused?`, `deleted?`, `success?`
-- `call` method pattern for service objects and scraper adapters
-- Private `set_*` for controller before_actions: `set_source`, `set_import_session`
-
-### Job Naming
-- Jobs suffixed with `Job`: `FetchFeedJob`, `ScrapeItemJob`, `ScheduleFetchesJob`
-- Queue assignment via `source_monitor_queue :fetch` class method
-
-### Controller Naming
-- Resource controllers follow Rails conventions: `SourcesController`, `ItemsController`
-- Singular nested resource controllers: `SourceFetchesController`, `SourceRetriesController`
-- Action-specific controllers for non-RESTful actions: `SourceBulkScrapesController`, `SourceHealthChecksController`
-
-## Configuration Patterns
-
-- Configuration via `SourceMonitor.configure { |config| ... }` block DSL
-- Nested configuration objects with `reset!` methods for testing
-- Settings classes with explicit defaults defined as constants
-- Callable values supported (procs/lambdas) for dynamic configuration
-
-## Error Handling
-
-- Custom error hierarchy: `FetchError` base with typed subclasses (`TimeoutError`, `ConnectionError`, `HTTPError`, `ParsingError`)
-- Errors carry structured data: `http_status`, `response`, `original_error`
-- Defensive `rescue StandardError` wrappers around logging calls
-- `Rails.logger` usage guarded with `defined?(Rails) && Rails.respond_to?(:logger) && Rails.logger`
-
-## Struct Usage
-
-- `Struct.new(..., keyword_init: true)` extensively used for result/event objects
-- Named structs: `Result`, `ResponseWrapper`, `EntryProcessingResult`
-- Event structs: `ItemCreatedEvent`, `ItemScrapedEvent`, `FetchCompletedEvent`
-
-## Frontend Conventions
-
-- CSS scoped under `.fm-admin` class with Tailwind `important` selector
-- Stimulus controllers registered on `window.SourceMonitorStimulus`
-- Controller naming: kebab-case in HTML (`async-submit`, `confirm-navigation`)
-- ERB partials prefixed with underscore, organized per resource
-- Turbo Stream responses via `SourceMonitor::TurboStreams::StreamResponder`
-
-## View Conventions
-
-- Layouts: Engine uses host app layout; views are ERB
-- Partials extensively used: `_row.html.erb`, `_details.html.erb`, `_form_fields.html.erb`
-- Wizard steps as separate partials: `steps/_upload.html.erb`, `steps/_preview.html.erb`
-- Toast notifications broadcast via realtime system
-
-## Test Conventions
-
-- Minitest with Rails test helpers (not RSpec)
-- Test files mirror source structure: `test/lib/source_monitor/scraping/item_scraper_test.rb`
-- Test helper methods: `create_source!` factory method in `ActiveSupport::TestCase`
-- Parallel test execution by default
-- Configuration reset in `setup` block: `SourceMonitor.reset_configuration!`
-- WebMock disables external connections
-- VCR for HTTP interaction recording
-- test-prof `before_all` for expensive shared setup
-- `with_inline_jobs` and `with_queue_adapter` helpers for job testing
-
-## Documentation Conventions
-
-- YARD-style comments in some service classes
-- Inline comments explain "why" not "what"
-- `# :nocov:` markers for untestable/defensive code paths
-- CHANGELOG maintained in conventional format
-- CONTRIBUTING.md present

data/.vbw-planning/codebase/DEPENDENCIES.md

@@ -1,100 +0,0 @@
-# Dependencies
-
-## Runtime Dependencies (gemspec)
-
-These are declared in `source_monitor.gemspec` and required by any host app using the engine.
-
-| Gem | Constraint | Purpose | Risk |
-|-----|-----------|---------|------|
-| rails | >= 8.0.3, < 9.0 | Core framework | Major version lock |
-| cssbundling-rails | ~> 1.4 | CSS asset bundling bridge | Low |
-| jsbundling-rails | ~> 1.3 | JS asset bundling bridge | Low |
-| turbo-rails | ~> 2.0 | Hotwire Turbo for real-time updates | Medium - optional but recommended |
-| feedjira | >= 3.2, < 5.0 | Feed parsing (RSS/Atom) | Core functionality |
-| faraday | ~> 2.9 | HTTP client | Core functionality |
-| faraday-retry | ~> 2.2 | Retry middleware | Low |
-| faraday-follow_redirects | ~> 0.4 | Redirect handling | Low |
-| faraday-gzip | ~> 3.0 | Compression support | Low |
-| nokolexbor | ~> 0.5 | Fast HTML parsing (lexbor engine) | Medium - native extension |
-| ruby-readability | ~> 0.7 | Article content extraction | Medium - older gem |
-| solid_queue | >= 0.3, < 3.0 | Background job processing | Core functionality |
-| solid_cable | >= 3.0, < 4.0 | Action Cable adapter | Real-time features |
-| ransack | ~> 4.2 | Search/filter form builder | Medium |
-
-## Optional Runtime Dependencies
-
-Loaded with rescue from LoadError in `lib/source_monitor.rb`:
-- `solid_queue` -- optional if host uses different Active Job backend
-- `solid_cable` -- optional if host uses Redis or another Action Cable adapter
-- `turbo-rails` -- optional but recommended for real-time updates
-- `ransack` -- powers search forms when available
-
-## Development Dependencies (Gemfile)
-
-| Gem | Purpose |
-|-----|---------|
-| puma | Development web server |
-| pg | PostgreSQL adapter |
-| propshaft | Asset pipeline |
-| rubocop-rails-omakase | Linting (Rails omakase style) |
-| brakeman | Security scanning |
-
-## Test Dependencies (Gemfile)
-
-| Gem | Purpose |
-|-----|---------|
-| simplecov | Code coverage reporting |
-| test-prof | Test profiling toolkit |
-| stackprof | Stack sampling profiler |
-| capybara | System test framework |
-| webmock | HTTP stubbing |
-| vcr | HTTP recording/playback |
-| selenium-webdriver | Browser driver for system tests |
-
-## JavaScript Dependencies (package.json)
-
-### Runtime
-
-| Package | Version | Purpose |
-|---------|---------|---------|
-| @hotwired/stimulus | ^3.2.2 | Stimulus JS framework |
-| stimulus-use | ^0.52.0 | Composable Stimulus behaviors |
-
-### Development
-
-| Package | Version | Purpose |
-|---------|---------|---------|
-| esbuild | ^0.23.0 | JS bundling |
-| tailwindcss | ^3.4.10 | CSS framework |
-| postcss | ^8.4.45 | CSS transformation |
-| autoprefixer | ^10.4.20 | CSS vendor prefixes |
-| eslint | ^9.11.0 | JS linting |
-| @eslint/js | ^9.11.0 | ESLint core config |
-| stylelint | ^16.8.0 | CSS linting |
-| stylelint-config-standard | ^36.0.0 | Stylelint standard config |
-
-## Dependency Coupling Analysis
-
-### Tight Coupling (hard to replace)
-- **Rails 8.x** -- entire engine is built on Rails conventions
-- **Feedjira** -- core feed parsing logic, deeply integrated in `Fetching::FeedFetcher`
-- **Faraday** -- HTTP client used throughout `SourceMonitor::HTTP`, configurable via middleware stack
-- **Solid Queue** -- integrated in engine initializer, scheduler, and job visibility system
-- **PostgreSQL** -- uses `FOR UPDATE SKIP LOCKED`, `NULLS FIRST/LAST` SQL syntax
-
-### Moderate Coupling (replaceable with effort)
-- **Nokolexbor/Nokogiri** -- HTML parsing in scrapers and OPML import
-- **Ransack** -- used in model `ransackable_attributes` declarations and search forms
-- **Tailwind CSS** -- all views use Tailwind utility classes scoped under `.fm-admin`
-
-### Loose Coupling (easily replaceable)
-- **Solid Cable** -- configurable via `config.realtime.adapter`, supports `:solid_cable`, `:redis`, `:async`
-- **ruby-readability** -- wrapped in `Scrapers::Readability` adapter behind pluggable adapter interface
-- **Turbo Rails** -- optional, loaded conditionally
-
-## Version Constraints of Note
-
-- Ruby >= 3.4.0 is a relatively aggressive minimum requirement
-- Rails >= 8.0.3 pins to the latest major, narrowing host app compatibility
-- Solid Queue has a wide range (0.3 to 3.0), suggesting early adoption with forward-looking flexibility
-- PostgreSQL is the only supported database (uses PG-specific SQL features)

data/.vbw-planning/codebase/INDEX.md

@@ -1,86 +0,0 @@
-# Index
-
-Cross-referenced index of key findings across all mapping documents.
-
-## Quick Reference
-
-| Document | Focus | Key Finding |
-|----------|-------|-------------|
-| [STACK.md](STACK.md) | Technology choices | Rails 8.1.1 engine, Ruby 3.4+, PostgreSQL, Solid Queue, Tailwind 3 |
-| [DEPENDENCIES.md](DEPENDENCIES.md) | Dependency analysis | 14 runtime gems, PG-only, optional deps loaded silently |
-| [ARCHITECTURE.md](ARCHITECTURE.md) | System design | 10 domain modules, event-driven, pluggable scrapers |
-| [STRUCTURE.md](STRUCTURE.md) | Directory layout | ~324 Ruby files, 124 tests, 24 migrations |
-| [CONVENTIONS.md](CONVENTIONS.md) | Code style | Rails omakase, frozen strings, Struct-based results |
-| [TESTING.md](TESTING.md) | Test infrastructure | Minitest, parallel, SimpleCov branch coverage, nightly profiling |
-| [CONCERNS.md](CONCERNS.md) | Risks & debt | Large files, PG lock-in, coverage gaps, no default auth |
-| [PATTERNS.md](PATTERNS.md) | Recurring patterns | Service objects, adapter pattern, event callbacks, Turbo Streams |
-
-## Key Entry Points
-
-| Purpose | File | Notes |
-|---------|------|-------|
-| Gem entry point | `lib/source_monitor.rb` | 102+ require statements, module definition |
-| Engine definition | `lib/source_monitor/engine.rb` | Initializers, asset registration |
-| Configuration DSL | `lib/source_monitor/configuration.rb` | 12 nested settings classes |
-| Routes | `config/routes.rb` | 24 lines, RESTful resources |
-| Main model | `app/models/source_monitor/source.rb` | Core domain entity |
-| Dashboard | `app/controllers/source_monitor/dashboard_controller.rb` | Landing page |
-| Fetch pipeline | `lib/source_monitor/fetching/feed_fetcher.rb` | Core data ingestion |
-| Scrape pipeline | `lib/source_monitor/scraping/item_scraper.rb` | Content extraction orchestrator |
-| Scheduler | `lib/source_monitor/scheduler.rb` | Periodic fetch scheduling |
-| JS entry | `app/assets/javascripts/source_monitor/application.js` | Stimulus app setup |
-| CSS entry | `app/assets/stylesheets/source_monitor/application.tailwind.css` | Tailwind input |
-| Test entry | `test/test_helper.rb` | Test infrastructure setup |
-
-## Data Model Reference
-
-| Model | Table | Key Relationships |
-|-------|-------|-------------------|
-| `Source` | `sourcemon_sources` | has_many: items, fetch_logs, scrape_logs, health_check_logs, log_entries |
-| `Item` | `sourcemon_items` | belongs_to: source; has_one: item_content; has_many: scrape_logs, log_entries |
-| `ItemContent` | `sourcemon_item_contents` | belongs_to: item (separate table for large scraped content) |
-| `FetchLog` | `sourcemon_fetch_logs` | belongs_to: source; has_one: log_entry (polymorphic) |
-| `ScrapeLog` | `sourcemon_scrape_logs` | belongs_to: item, source; has_one: log_entry (polymorphic) |
-| `HealthCheckLog` | `sourcemon_health_check_logs` | belongs_to: source; has_one: log_entry (polymorphic) |
-| `LogEntry` | `sourcemon_log_entries` | delegated_type: loggable (FetchLog/ScrapeLog/HealthCheckLog) |
-| `ImportSession` | `sourcemon_import_sessions` | JSONB state for wizard flow |
-| `ImportHistory` | `sourcemon_import_histories` | Records completed imports |
-
-## Job Reference
-
-| Job Class | Queue | Schedule | Purpose |
-|-----------|-------|----------|---------|
-| `ScheduleFetchesJob` | fetch | Recurring | Triggers scheduler to find due sources |
-| `FetchFeedJob` | fetch | On-demand | Fetches one source's feed |
-| `ScrapeItemJob` | scrape | On-demand | Scrapes one item's content |
-| `SourceHealthCheckJob` | fetch | On-demand | Health check for one source |
-| `ImportSessionHealthCheckJob` | fetch | On-demand | Health check during OPML import |
-| `ImportOpmlJob` | fetch | On-demand | Bulk creates sources from OPML |
-| `LogCleanupJob` | fetch | Recurring | Prunes old log entries |
-| `ItemCleanupJob` | fetch | Recurring | Prunes items per retention policy |
-
-## Configuration Surface Area
-
-| Section | Key Settings | Defaults |
-|---------|-------------|----------|
-| Queues | `fetch_queue_name`, `scrape_queue_name`, concurrency | `source_monitor_fetch`, `source_monitor_scrape`, 2 each |
-| HTTP | timeout, retries, user agent, proxy, headers | 15s/5s timeout, 4 retries |
-| Fetching | adaptive interval params, jitter | 5min-24hr, 1.25x increase, 0.75x decrease |
-| Health | window size, thresholds, auto-pause | 20 window, 0.8/0.5/0.2 thresholds |
-| Scraping | max_in_flight, max_bulk_batch | 25, 100 |
-| Retention | days, max_items, strategy | nil (no auto-cleanup), :destroy |
-| Realtime | adapter (solid_cable/redis/async) | solid_cable |
-| Authentication | handlers, current_user_method | nil (no auth by default) |
-| Models | table_name_prefix, concerns, validations | `sourcemon_` |
-
-## Critical Cross-Cutting Concerns
-
-1. **PG-only** (ARCHITECTURE + CONCERNS): `FOR UPDATE SKIP LOCKED` and `NULLS FIRST/LAST` SQL are PostgreSQL-specific. No other DB supported.
-
-2. **No default auth** (ARCHITECTURE + CONCERNS): Engine mounts without authentication unless host app configures it. Import wizard has a `create_guest_user` fallback.
-
-3. **Eager loading** (STRUCTURE + CONCERNS): All 102+ require statements in `lib/source_monitor.rb` load at boot time.
-
-4. **Coverage debt** (TESTING + CONCERNS): `config/coverage_baseline.json` lists 2329 lines of known uncovered code, particularly in `FeedFetcher`, `ItemCreator`, `Configuration`, and `Dashboard::Queries`.
-
-5. **Large files** (STRUCTURE + CONCERNS): `FeedFetcher` (627 lines), `Configuration` (655 lines), and `ImportSessionsController` (792 lines) are candidates for extraction.

data/.vbw-planning/codebase/META.md

@@ -1,42 +0,0 @@
-# META
-
-## Mapping Metadata
-
-| Field | Value |
-|-------|-------|
-| mapped_at | 2026-02-09T00:00:00Z |
-| git_hash | 1fb781a35575c2d46bfb8cd96f90c64ddf6ed210 |
-| file_count | 530 |
-| mode | full |
-| monorepo | false |
-| mapping_tier | solo (inline) |
-
-## Documents
-
-| File | Domain | Description |
-|------|--------|-------------|
-| `STACK.md` | Tech Stack | Technology choices, framework versions, build pipeline |
-| `DEPENDENCIES.md` | Tech Stack | Runtime, dev, and test dependency analysis with coupling assessment |
-| `ARCHITECTURE.md` | Architecture | System overview, domain modules, data model, job architecture |
-| `STRUCTURE.md` | Architecture | Full directory tree with file counts and organization |
-| `CONVENTIONS.md` | Quality | Naming conventions, code style, frontend patterns |
-| `TESTING.md` | Quality | Test framework, CI pipeline, coverage strategy, profiling |
-| `CONCERNS.md` | Concerns | Technical debt, security risks, performance, operational risks |
-| `INDEX.md` | Synthesis | Cross-referenced index with key findings and quick references |
-| `PATTERNS.md` | Synthesis | 14 recurring patterns across the codebase |
-| `META.md` | Meta | This file -- mapping metadata and document inventory |
-
-## Language Breakdown
-
-| Language | File Count | Notes |
-|----------|-----------|-------|
-| Ruby (.rb) | ~324 | Models, controllers, jobs, lib modules, tests |
-| ERB (.erb) | ~48 | View templates |
-| Markdown (.md) | ~45 | Documentation |
-| YAML (.yml) | ~16 | Configuration files |
-| JavaScript (.js) | ~14 | Stimulus controllers, build entry points |
-| CSS (.css) | ~2 | Tailwind input and build output |
-
-## Project Summary
-
-SourceMonitor is a mountable Rails 8 engine (v0.2.1) that ingests RSS/Atom/JSON feeds, scrapes full article content via pluggable adapters, and provides Solid Queue-powered dashboards for monitoring and remediation. It is distributed as a RubyGem, requires PostgreSQL and Ruby >= 3.4.0, and integrates with host Rails applications via the standard engine mounting pattern with an isolated namespace.