source_monitor 0.7.1 → 0.8.1

data/.vbw-planning/codebase/STRUCTURE.md

@@ -1,324 +0,0 @@
-# Directory Structure
-
-## Top-Level Layout
-
-```
-source_monitor/
-  app/                    # Rails engine application code
-  bin/                    # Scripts (rubocop, test runners, CI checks)
-  config/                 # Engine configuration (routes, tailwind, feedjira initializer)
-  db/migrate/             # Engine migrations (24 migration files)
-  docs/                   # Documentation
-  examples/               # Example configurations and adapters
-  lib/                    # Engine library code and rake tasks
-  test/                   # Test suite
-  tasks/                  # Rake task definitions (aliases)
-  coverage/               # SimpleCov output (gitignored content)
-  node_modules/           # NPM packages
-  pkg/                    # Gem build output
-  tmp/                    # Temporary files
-```
-
-## App Directory (`app/`)
-
-```
-app/
-  assets/
-    builds/source_monitor/     # Pre-built CSS and JS (committed)
-      application.css          # Built Tailwind CSS output
-      application.js           # Built esbuild JS output
-    config/
-      source_monitor_manifest.js
-    images/source_monitor/     # SVGs and icons
-    javascripts/source_monitor/
-      application.js           # Stimulus app entry point
-      turbo_actions.js         # Custom Turbo Stream actions
-      controllers/
-        async_submit_controller.js
-        confirm_navigation_controller.js
-        dropdown_controller.js
-        modal_controller.js
-        notification_controller.js
-        select_all_controller.js
-    stylesheets/source_monitor/
-      application.tailwind.css # Tailwind input file
-    svgs/source_monitor/       # SVG assets
-  controllers/source_monitor/
-    application_controller.rb
-    dashboard_controller.rb
-    fetch_logs_controller.rb
-    health_controller.rb
-    import_sessions_controller.rb
-    items_controller.rb
-    logs_controller.rb
-    scrape_logs_controller.rb
-    source_bulk_scrapes_controller.rb
-    source_fetches_controller.rb
-    source_health_checks_controller.rb
-    source_health_resets_controller.rb
-    source_retries_controller.rb
-    source_turbo_responses.rb
-    sources_controller.rb
-    concerns/
-      sanitizes_search_params.rb
-  jobs/source_monitor/
-    application_job.rb
-    fetch_feed_job.rb
-    import_opml_job.rb
-    import_session_health_check_job.rb
-    item_cleanup_job.rb
-    log_cleanup_job.rb
-    schedule_fetches_job.rb
-    scrape_item_job.rb
-    source_health_check_job.rb
-  mailers/source_monitor/
-    application_mailer.rb
-  models/source_monitor/
-    application_record.rb
-    fetch_log.rb
-    health_check_log.rb
-    import_history.rb
-    import_session.rb
-    item.rb
-    item_content.rb
-    log_entry.rb
-    scrape_log.rb
-    source.rb
-    concerns/
-      loggable.rb
-  views/source_monitor/
-    dashboard/
-      index.html.erb
-      _fetch_schedule.html.erb
-      _job_metrics.html.erb
-      _recent_activity.html.erb
-      _stat_card.html.erb
-      _stats.html.erb
-    fetch_logs/
-      show.html.erb
-    import_sessions/
-      show.html.erb
-      show.turbo_stream.erb
-      _header.html.erb
-      _sidebar.html.erb
-      health_check/
-        _progress.html.erb
-        _row.html.erb
-      steps/
-        _configure.html.erb
-        _confirm.html.erb
-        _health_check.html.erb
-        _navigation.html.erb
-        _preview.html.erb
-        _upload.html.erb
-    items/
-      index.html.erb
-      show.html.erb
-      _details.html.erb
-      _details_wrapper.html.erb
-    logs/
-      index.html.erb
-    scrape_logs/
-      show.html.erb
-    shared/
-      _toast.html.erb
-    sources/
-      index.html.erb
-      show.html.erb
-      new.html.erb
-      edit.html.erb
-      _bulk_scrape_form.html.erb
-      _bulk_scrape_modal.html.erb
-      _details.html.erb
-      _details_wrapper.html.erb
-      _empty_state_row.html.erb
-      _fetch_interval_heatmap.html.erb
-      _form.html.erb
-      _form_fields.html.erb
-      _health_status_badge.html.erb
-      _import_history_panel.html.erb
-      _row.html.erb
-```
-
-## Lib Directory (`lib/source_monitor/`)
-
-```
-lib/
-  source_monitor.rb               # Main entry point, requires, module definition
-  source_monitor/
-    version.rb                    # VERSION constant (0.2.1)
-    engine.rb                     # Rails::Engine with initializers
-    configuration.rb              # Configuration DSL (655 lines)
-    events.rb                     # Event system (dispatch, callbacks)
-    instrumentation.rb            # ActiveSupport::Notifications wrapper
-    metrics.rb                    # In-memory counters/gauges
-    http.rb                       # Faraday client builder
-    model_extensions.rb           # Dynamic model concern/validation injection
-    scheduler.rb                  # Source fetch scheduling with SKIP LOCKED
-    health.rb                     # Health module setup
-    realtime.rb                   # Realtime broadcasting setup
-    feedjira_extensions.rb        # Feedjira customizations
-    assets.rb                     # Asset management utilities
-
-    analytics/                    # Query objects for dashboard metrics
-      source_activity_rates.rb
-      source_fetch_interval_distribution.rb
-      sources_index_metrics.rb
-    assets/                       # Asset bundling helpers
-    dashboard/                    # Dashboard presenters and queries
-      queries.rb
-      quick_action.rb
-      quick_actions_presenter.rb
-      recent_activity.rb
-      recent_activity_presenter.rb
-      turbo_broadcaster.rb
-      upcoming_fetch_schedule.rb
-    fetching/                     # Feed fetching pipeline
-      feed_fetcher.rb             # Core fetcher (627 lines)
-      fetch_error.rb
-      fetch_runner.rb
-      retry_policy.rb
-      stalled_fetch_reconciler.rb
-    health/                       # Health monitoring
-      import_source_health_check.rb
-      source_health_check.rb
-      source_health_monitor.rb
-      source_health_reset.rb
-    import_sessions/              # OPML import support
-      entry_normalizer.rb
-    items/                        # Item management
-      item_creator.rb
-      retention_pruner.rb
-    jobs/                         # Job support modules
-      cleanup_options.rb
-      fetch_failure_subscriber.rb
-      solid_queue_metrics.rb
-      visibility.rb
-    logs/                         # Unified log system
-      entry_sync.rb
-      filter_set.rb
-      query.rb
-      table_presenter.rb
-    models/                       # Shared model concerns
-      sanitizable.rb
-      url_normalizable.rb
-    pagination/                   # Pagination support
-      paginator.rb
-    realtime/                     # Realtime broadcasting
-      adapter.rb
-      broadcaster.rb
-    release/                      # Release management
-      changelog.rb
-      runner.rb
-    scrapers/                     # Scraper adapters
-      base.rb
-      readability.rb
-      fetchers/
-        http_fetcher.rb
-      parsers/
-        readability_parser.rb
-    scraping/                     # Scraping orchestration
-      bulk_result_presenter.rb
-      bulk_source_scraper.rb
-      enqueuer.rb
-      item_scraper.rb
-      item_scraper/
-        adapter_resolver.rb
-        persistence.rb
-      scheduler.rb
-      state.rb
-    security/                     # Security modules
-      authentication.rb
-      parameter_sanitizer.rb
-    setup/                        # Installation workflow
-      bundle_installer.rb
-      cli.rb
-      dependency_checker.rb
-      detectors.rb
-      gemfile_editor.rb
-      initializer_patcher.rb
-      install_generator.rb
-      migration_installer.rb
-      node_installer.rb
-      prompter.rb
-      requirements.rb
-      shell_runner.rb
-      workflow.rb
-      verification/
-        action_cable_verifier.rb
-        printer.rb
-        result.rb
-        runner.rb
-        solid_queue_verifier.rb
-        telemetry_logger.rb
-    sources/                      # Source-specific support
-      params.rb
-      turbo_stream_presenter.rb
-    turbo_streams/                # Turbo Stream helpers
-      stream_responder.rb
-  tasks/                          # Rake tasks
-    recover_stalled_fetches.rake
-    source_monitor_assets.rake
-    source_monitor_setup.rake
-    source_monitor_tasks.rake
-    test_smoke.rake
-```
-
-## Test Directory (`test/`)
-
-```
-test/
-  test_helper.rb                  # Test configuration and shared helpers
-  test_prof.rb                    # test-prof integration
-  source_monitor_test.rb          # Module-level tests
-  fixtures/                       # Test fixtures
-    vcr_cassettes/                # VCR recorded HTTP interactions
-  dummy/                          # Full Rails dummy app for testing
-    app/
-    bin/
-    config/
-    db/
-  examples/                       # Example integration tests
-    advanced_template_test.rb
-    basic_template_test.rb
-    custom_adapter_example_test.rb
-    docker_config_test.rb
-  integration/                    # Integration tests
-    engine_mounting_test.rb
-    host_install_flow_test.rb
-    navigation_test.rb
-    release_packaging_test.rb
-  lib/source_monitor/             # Unit tests mirroring lib/ structure
-    configuration_test.rb
-    feedjira_configuration_test.rb
-    instrumentation_test.rb
-    health/
-    pagination/
-    release/
-    scraping/
-    security/  (implicitly tested)
-    setup/
-    turbo_streams/
-  mailers/
-  models/source_monitor/          # Model tests
-  system/                         # System/browser tests
-    dashboard_test.rb
-    dropdown_fallback_test.rb
-    items_test.rb
-    logs_test.rb
-    mission_control_test.rb
-    sources_test.rb
-  tasks/                          # Rake task tests
-```
-
-## Key File Counts
-
-| Category | Count |
-|----------|-------|
-| Ruby files (.rb) | ~324 |
-| ERB templates (.erb) | ~48 |
-| JavaScript files (.js) | ~14 |
-| YAML configs (.yml) | ~16 |
-| Test files (*_test.rb) | ~124 |
-| Migrations | 24 |
-| Stimulus controllers | 6 |

data/.vbw-planning/codebase/TESTING.md

@@ -1,154 +0,0 @@
-# Testing
-
-## Framework & Tools
-
-- **Test Framework**: Minitest (Rails default)
-- **System Tests**: Capybara + Selenium WebDriver (Chrome)
-- **HTTP Mocking**: WebMock (disables all external connections) + VCR (recorded cassettes)
-- **Coverage**: SimpleCov with branch coverage enabled
-- **Profiling**: test-prof (TagProf, EventProf) + StackProf
-- **Parallelization**: Built-in Rails parallel testing (configurable workers via `SOURCE_MONITOR_TEST_WORKERS`)
-
-## Test Infrastructure
-
-### Test Helper (`test/test_helper.rb`)
-- Loads SimpleCov in CI or when `COVERAGE` env is set
-- Configures Rails test environment pointing to dummy app
-- Sets migration paths to include both engine and dummy migrations
-- Uses `:test` ActiveJob queue adapter by default
-- Fixtures loaded from `test/fixtures/`
-- VCR cassettes stored in `test/vcr_cassettes/`
-- WebMock allows localhost only
-- Random test ordering enabled
-- Configuration reset in every test via `SourceMonitor.reset_configuration!`
-
-### test-prof Integration (`test/test_prof.rb`)
-- `TestProf::BeforeAll::Minitest` for `before_all` blocks (shared expensive setup)
-- `SourceMonitor::TestProfSupport::SetupOnce` -- `setup_once` alias for `before_all`
-- `SourceMonitor::TestProfSupport::InlineJobs` -- `with_inline_jobs` helper
-- `TestProf::MinitestSample` -- SAMPLE/SAMPLE_GROUPS env var support for focused runs
-
-### Shared Test Helpers
-- `create_source!(attributes = {})` -- factory method for creating test sources
-- `with_queue_adapter(adapter)` -- temporarily switch ActiveJob adapter
-
-## Test Categories
-
-| Category | File Count | Path | Purpose |
-|----------|-----------|------|---------|
-| Unit (lib) | ~75 | `test/lib/source_monitor/` | Lib module tests |
-| Model | ~10 | `test/models/source_monitor/` | Model validation, scopes, behavior |
-| System | 6 | `test/system/` | Browser-driven end-to-end tests |
-| Integration | 4 | `test/integration/` | Engine mounting, navigation, packaging |
-| Example | 4 | `test/examples/` | Template and adapter examples |
-| Task | 2 | `test/tasks/` | Rake task tests |
-| Mailer | 1 | `test/mailers/` | Application mailer test |
-| Module | 1 | `test/source_monitor_test.rb` | Top-level module tests |
-
-**Total: ~124 test files**
-
-## Test Structure
-
-Tests mirror the source directory structure:
-
-```
-test/lib/source_monitor/
-  configuration_test.rb
-  feedjira_configuration_test.rb
-  instrumentation_test.rb
-  health/
-    health_module_test.rb
-    source_health_check_test.rb
-    source_health_monitor_test.rb
-    source_health_reset_test.rb
-  pagination/
-    paginator_test.rb
-  scraping/
-    bulk_result_presenter_test.rb
-    bulk_source_scraper_test.rb
-    enqueuer_test.rb
-    item_scraper_test.rb
-    item_scraper/
-      adapter_resolver_test.rb
-      persistence_test.rb
-    scheduler_test.rb
-    state_test.rb
-  setup/
-    bundle_installer_test.rb
-    cli_test.rb
-    dependency_checker_test.rb
-    detectors_test.rb
-    gemfile_editor_test.rb
-    initializer_patcher_test.rb
-    install_generator_test.rb
-    migration_installer_test.rb
-    node_installer_test.rb
-    prompter_test.rb
-    requirements_test.rb
-    workflow_test.rb
-    verification/
-      action_cable_verifier_test.rb
-      printer_test.rb
-      runner_test.rb
-      solid_queue_verifier_test.rb
-      telemetry_logger_test.rb
-  turbo_streams/
-    stream_responder_test.rb
-  release/
-    runner_test.rb
-```
-
-## Dummy Application (`test/dummy/`)
-
-Full Rails application used as the host app for testing:
-- PostgreSQL database (`config/database.yml`)
-- Solid Queue configuration (`config/solid_queue.yml`, `config/queue.yml`)
-- Solid Cable configuration (`config/cable.yml`)
-- Mission Control integration
-- Custom STI model: `SourceMonitor::SponsoredSource` (tests model extensions)
-- Extension concern: `DummySourceMonitor::SourceExtensions`
-- `User` model for testing authentication
-- Source Monitor initializer exercising the full configuration API
-
-## CI Pipeline
-
-### Test Job
-1. Sets up PostgreSQL 15 service container
-2. Installs Ruby 3.4.4, Node 20
-3. Builds frontend assets (`npm run build`)
-4. Creates and migrates test database
-5. Runs full test suite with coverage (`bin/test-coverage`)
-6. Enforces diff coverage (`bin/check-diff-coverage`)
-7. Uploads coverage artifact
-8. Captures system test screenshots on failure
-
-### Release Verification Job
-- Depends on lint, security, test jobs
-- Runs `test/integration/release_packaging_test.rb` specifically
-- Enforces diff coverage again
-
-### Profiling Job (Nightly)
-- Runs on schedule (`cron: "30 6 * * *"`)
-- TagProf by type
-- EventProf on `sql.active_record`
-- StackProf on integration tests
-- Enforces profiling guardrails (`bin/check-test-prof-metrics`)
-- Uploads profiling artifacts
-
-## Coverage
-
-- Branch coverage enabled
-- `refuse_coverage_drop :line` prevents regressions
-- Coverage baseline tracked in `config/coverage_baseline.json` (lists uncovered lines per file)
-- Diff coverage enforcement ensures new code is tested
-- `SOURCE_MONITOR_SKIP_COVERAGE` env var to disable coverage collection
-- `# :nocov:` annotations used for defensive/fallback code paths
-
-## Notable Testing Patterns
-
-- **Configuration Reset**: Every test resets `SourceMonitor.reset_configuration!` in `setup`
-- **WebMock**: All external HTTP disabled; tests use stubs or VCR cassettes
-- **Job Testing**: Tests use `:test` adapter; `with_inline_jobs` for synchronous execution
-- **Parallel Safety**: Tests designed for parallel execution; use `SecureRandom.hex` for unique fixtures
-- **Factory Method**: `create_source!` with `save!(validate: false)` for flexible test data
-- **Setup Verification Tests**: `bin/check-setup-tests` ensures all setup files have corresponding tests

data/.vbw-planning/config.json

@@ -1,53 +0,0 @@
-{
-  "effort": "thorough",
-  "autonomy": "standard",
-  "auto_commit": true,
-  "planning_tracking": "manual",
-  "auto_push": "never",
-  "verification_tier": "standard",
-  "skill_suggestions": true,
-  "auto_install_skills": false,
-  "discovery_questions": true,
-  "context_compiler": true,
-  "visual_format": "unicode",
-  "max_tasks_per_plan": 5,
-  "prefer_teams": "always",
-  "branch_per_milestone": false,
-  "plain_summary": true,
-  "active_profile": "default",
-  "custom_profiles": {},
-  "model_profile": "quality",
-  "model_overrides": {},
-  "agent_max_turns": {
-    "scout": 15,
-    "qa": 25,
-    "architect": 30,
-    "debugger": 80,
-    "lead": 50,
-    "dev": 75
-  },
-  "qa_skip_agents": [
-    "docs"
-  ],
-  "v3_delta_context": false,
-  "v3_context_cache": false,
-  "v3_plan_research_persist": false,
-  "v3_metrics": false,
-  "v3_contract_lite": false,
-  "v3_lock_lite": false,
-  "v3_validation_gates": false,
-  "v3_smart_routing": false,
-  "v3_event_log": false,
-  "v3_schema_validation": false,
-  "v3_snapshot_resume": false,
-  "v3_lease_locks": false,
-  "v3_event_recovery": false,
-  "v3_monorepo_routing": false,
-  "v2_hard_contracts": false,
-  "v2_hard_gates": false,
-  "v2_typed_protocol": false,
-  "v2_role_isolation": false,
-  "v2_two_phase_completion": false,
-  "v2_token_budgets": false,
-  "compaction_trigger": 130000
-}

data/.vbw-planning/discovery.json

@@ -1,26 +0,0 @@
-{
-  "answered": [
-    {
-      "question": "What matters most in the conventions cleanup?",
-      "answer": "All of the above: Model conventions, Controller patterns, Dead code removal",
-      "category": "scope",
-      "phase": "4",
-      "date": "2026-02-10"
-    },
-    {
-      "question": "How should we handle convention violations that would change public API behavior?",
-      "answer": "Fix everything -- rename/restructure even if it changes method signatures or route patterns",
-      "category": "api-policy",
-      "phase": "4",
-      "date": "2026-02-10"
-    }
-  ],
-  "inferred": [
-    "User wants comprehensive cleanup, not surface-level",
-    "Public API changes are acceptable for convention alignment",
-    "Tests should be updated to match any changes, not removed",
-    "All layers: models, controllers, services, dead code",
-    "Generator should be maximally helpful -- create files rather than just warn",
-    "Target Rails 8 defaults (queue.yml) rather than supporting legacy naming"
-  ]
-}

data/.vbw-planning/milestones/default/ROADMAP.md

@@ -1,115 +0,0 @@
-<!-- VBW ROADMAP TEMPLATE (ARTF-07) -- Phase-based project roadmap -->
-<!-- Created by Architect agent during /vbw scope -->
-
-# SourceMonitor Roadmap
-
-## Overview
-
-This roadmap focuses on stabilizing and improving the existing SourceMonitor codebase through test coverage improvements and refactoring for maintainability. 4 phases progressing from analysis to coverage to refactoring to cleanup.
-
-## Phases
-
-- [x] Phase 1: Coverage Analysis & Quick Wins
-- [x] Phase 2: Critical Path Test Coverage
-- [x] Phase 3: Large File Refactoring
-- [x] Phase 4: Code Quality & Conventions Cleanup
-
-## Phase Details
-
-### Phase 1: Coverage Analysis & Quick Wins
-
-**Goal:** Analyze the coverage baseline, identify the highest-impact gaps, and close easy coverage wins (frozen_string_literal, small utility classes).
-**Depends on:** None
-
-**Requirements:**
-- REQ-13: Frozen string literal consistency
-- REQ-14: RuboCop audit
-
-**Success Criteria:**
-1. All Ruby files have frozen_string_literal: true
-2. Zero RuboCop violations against omakase ruleset
-3. Coverage baseline shrinks by at least 10%
-
-**Plans:**
-- [x] Plan 01: frozen-string-literal-audit (5f02db8)
-- [x] Plan 02: rubocop-audit-and-fix (no changes needed)
-
-### Phase 2: Critical Path Test Coverage
-
-**Goal:** Close the major test coverage gaps in the most critical business logic -- feed fetching, item creation, configuration, dashboard queries, and broadcasting.
-**Depends on:** Phase 1
-
-**Requirements:**
-- REQ-01: FeedFetcher coverage
-- REQ-02: ItemCreator coverage
-- REQ-03: Configuration coverage
-- REQ-04: Dashboard::Queries coverage
-- REQ-05: Broadcaster coverage
-- REQ-06: BulkSourceScraper coverage
-- REQ-07: SourcesIndexMetrics coverage
-
-**Success Criteria:**
-1. Coverage baseline shrinks by at least 50% from original
-2. All critical path files have branch coverage above 80%
-3. CI pipeline passes with no regressions
-
-**Plans:**
-- [x] Plan 01: feed-fetcher-tests (8d4e8d3)
-- [x] Plan 02: item-creator-tests (ce8ede4)
-- [x] Plan 03: configuration-tests (66b8df2)
-- [x] Plan 04: dashboard-and-analytics-tests (a8f2611, 2e50580)
-- [x] Plan 05: scraping-and-broadcasting-tests (e497891, 66b8df2)
-
-### Phase 3: Large File Refactoring
-
-**Goal:** Break down the three largest files (FeedFetcher, Configuration, ImportSessionsController) into focused, single-responsibility modules while maintaining all existing tests.
-**Depends on:** Phase 2
-
-**Requirements:**
-- REQ-08: Extract FeedFetcher
-- REQ-09: Extract Configuration
-- REQ-10: Extract ImportSessionsController
-- REQ-11: Fix LogEntry table name
-- REQ-12: Replace eager requires with autoloading
-
-**Success Criteria:**
-1. No single file exceeds 300 lines
-2. All existing tests pass without modification (or with minimal adapter changes)
-3. Public API remains unchanged
-
-**Plans:**
-- [x] Plan 01: extract-feed-fetcher (2f00274)
-- [x] Plan 02: extract-configuration-settings (ab823a3)
-- [x] Plan 03: extract-import-sessions-controller (9dce996)
-- [x] Plan 04: fix-log-entry-and-autoloading (fb99d3d)
-
-### Phase 4: Code Quality & Conventions Cleanup
-
-**Goal:** Final pass to ensure all code follows Rails best practices and conventions, clean up any remaining debt.
-**Depends on:** Phase 3
-
-**Requirements:**
-- REQ-15: Rails conventions audit
-
-**Success Criteria:**
-1. All models, controllers, and service objects follow established conventions
-2. No RuboCop violations
-3. Coverage baseline is at least 60% smaller than original
-4. CI pipeline fully green
-
-**Plans:**
-- [x] Plan 01: conventions-audit (44fe6b6, c30a503, f070ea6, 78600b5, ec67c65)
-- [x] Plan 02: item-creator-extraction (3d33a71)
-- [x] Plan 03: final-verification (cc2e538)
-
-## Progress
-
-| Phase | Plans Complete | Status | Completed |
-|-------|---------------|--------|-----------|
-| 1 - Coverage Analysis & Quick Wins | 2/2 | complete | 2026-02-09 |
-| 2 - Critical Path Test Coverage | 5/5 | complete | 2026-02-09 |
-| 3 - Large File Refactoring | 4/4 | complete | 2026-02-10 |
-| 4 - Code Quality & Conventions Cleanup | 3/3 | complete | 2026-02-10 |
-
----
-*Last updated: 2026-02-10 after Phase 4 completion*