source_monitor 0.3.3 → 0.4.0

data/.vbw-planning/phases/05-active-storage-images/05-VERIFICATION.md

@@ -0,0 +1,156 @@
+---
+phase: 5
+tier: standard
+result: PASS
+passed: 34
+failed: 0
+total: 34
+date: 2026-02-12
+---
+
+# Phase 5 Verification: Active Storage Image Downloads
+
+## Must-Have Checks
+
+### PLAN-01: images-config-model-rewriter
+
+| # | Truth/Condition | Status | Evidence |
+|---|-----------------|--------|----------|
+| 1 | `PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/configuration/images_settings_test.rb` exits 0 with 0 failures | PASS | 15 runs, 21 assertions, 0 failures |
+| 2 | `PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/images/content_rewriter_test.rb` exits 0 with 0 failures | PASS | 27 runs, 41 assertions, 0 failures |
+| 3 | `PARALLEL_WORKERS=1 bin/rails test test/models/source_monitor/item_content_test.rb` exits 0 with 0 failures | PASS | 6 runs, 10 assertions, 0 failures |
+| 4 | `bin/rubocop` on PLAN-01 files exits 0 with no offenses | PASS | 3 files inspected, 0 offenses |
+| 5 | `SourceMonitor.config.images` returns an ImagesSettings instance | PASS | attr_reader :images in configuration.rb, @images = ImagesSettings.new |
+| 6 | `SourceMonitor.config.images.download_to_active_storage` defaults to `false` | PASS | DEFAULT in ImagesSettings.reset! sets to false |
+| 7 | `SourceMonitor.reset_configuration!` resets images settings to defaults | PASS | Tested in images_settings_test.rb (15 tests pass) |
+| 8 | ContentRewriter.new(html).image_urls returns an array of absolute image URLs | PASS | ContentRewriter#image_urls method verified (27 tests pass) |
+| 9 | ContentRewriter.new(html).rewrite { \|url\| new_url } replaces img src attributes | PASS | ContentRewriter#rewrite method verified (27 tests pass) |
+| 10 | ItemContent responds to `images` (has_many_attached) | PASS | has_many_attached :images in item_content.rb (6 tests pass) |
+
+### PLAN-02: download-job-integration-docs
+
+| # | Truth/Condition | Status | Evidence |
+|---|-----------------|--------|----------|
+| 11 | `PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/images/downloader_test.rb` exits 0 with 0 failures | PASS | 11 runs, 18 assertions, 0 failures |
+| 12 | `PARALLEL_WORKERS=1 bin/rails test test/jobs/source_monitor/download_content_images_job_test.rb` exits 0 with 0 failures | PASS | 10 runs, 29 assertions, 0 failures |
+| 13 | `PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/fetching/feed_fetcher/entry_processor_test.rb` exits 0 with 0 failures | PASS | 5 runs, 8 assertions, 0 failures |
+| 14 | `bin/rubocop` on PLAN-02 files exits 0 with no offenses | PASS | 3 files inspected, 0 offenses |
+| 15 | When `config.images.download_to_active_storage` is false (default), no job enqueued | PASS | entry_processor_test.rb verifies default behavior |
+| 16 | When `config.images.download_to_active_storage` is true, job enqueued for new items with HTML | PASS | entry_processor_test.rb verifies enabled behavior |
+| 17 | DownloadContentImagesJob downloads images, attaches to ItemContent, rewrites item.content | PASS | download_content_images_job_test.rb (10 tests verify full pipeline) |
+| 18 | Images that fail to download preserve original URLs (graceful fallback) | PASS | download_content_images_job_test.rb tests individual failure handling |
+| 19 | Images larger than max_download_size are skipped | PASS | downloader_test.rb tests size validation |
+| 20 | Images with disallowed content types are skipped | PASS | downloader_test.rb tests content type validation |
+| 21 | sm-configure skill documents the new config.images section | PASS | config.images in SKILL.md and configuration-reference.md |
+
+## Artifact Checks
+
+| Artifact | Exists | Contains | Status |
+|----------|--------|----------|--------|
+| `lib/source_monitor/configuration/images_settings.rb` | YES | class ImagesSettings | PASS |
+| `lib/source_monitor/images/content_rewriter.rb` | YES | class ContentRewriter | PASS |
+| `test/lib/source_monitor/configuration/images_settings_test.rb` | YES | class ImagesSettingsTest (15 tests) | PASS |
+| `test/lib/source_monitor/images/content_rewriter_test.rb` | YES | class ContentRewriterTest (27 tests) | PASS |
+| `test/models/source_monitor/item_content_test.rb` | YES | 6 tests for has_many_attached | PASS |
+| `lib/source_monitor/images/downloader.rb` | YES | class Downloader | PASS |
+| `app/jobs/source_monitor/download_content_images_job.rb` | YES | class DownloadContentImagesJob | PASS |
+| `test/lib/source_monitor/images/downloader_test.rb` | YES | 11 tests for Downloader | PASS |
+| `test/jobs/source_monitor/download_content_images_job_test.rb` | YES | 10 tests for job | PASS |
+| `test/lib/source_monitor/fetching/feed_fetcher/entry_processor_test.rb` | YES | 5 tests for integration hook | PASS |
+| `.claude/skills/sm-configure/SKILL.md` | YES | config.images section | PASS |
+| `.claude/skills/sm-configure/reference/configuration-reference.md` | YES | ImagesSettings documentation | PASS |
+
+## Key Link Checks
+
+| From | To | Via | Status |
+|------|----|----|--------|
+| images_settings.rb#download_to_active_storage | REQ-24 | Configurable option defaults to false | PASS |
+| content_rewriter.rb#image_urls | REQ-24 | Detects inline images in item content | PASS |
+| content_rewriter.rb#rewrite | REQ-24 | Replaces original URLs with Active Storage URLs | PASS |
+| item_content.rb#has_many_attached | REQ-24 | Images attached to ItemContent via Active Storage | PASS |
+| download_content_images_job.rb#perform | REQ-24 | Downloads inline images to Active Storage | PASS |
+| entry_processor.rb#enqueue_image_download | REQ-24 | Enqueues download job when config enabled | PASS |
+| downloader.rb | REQ-24 | Validates size and content type before download | PASS |
+| sm-configure/SKILL.md | REQ-24 | Configuration documented in skill | PASS |
+
+## Convention Compliance
+
+| Convention | File | Status | Detail |
+|------------|------|--------|--------|
+| frozen_string_literal | images_settings.rb | PASS | Present in line 1 |
+| frozen_string_literal | content_rewriter.rb | PASS | Present in line 1 |
+| frozen_string_literal | downloader.rb | PASS | Present in line 1 |
+| frozen_string_literal | download_content_images_job.rb | PASS | Present in line 1 |
+| Test coverage | ImagesSettings | PASS | 15 tests covering defaults, accessors, reset, download_enabled? |
+| Test coverage | ContentRewriter | PASS | 27 tests covering extraction, rewriting, edge cases |
+| Test coverage | Downloader | PASS | 11 tests covering success/failure modes |
+| Test coverage | DownloadContentImagesJob | PASS | 10 tests covering full pipeline and failures |
+| Test coverage | EntryProcessor integration | PASS | 5 tests covering enabled/disabled/failure modes |
+| Naming | ImagesSettings | PASS | Follows Configuration sub-class pattern |
+| Naming | Images module | PASS | Follows engine module structure |
+| Autoloading | Images::ContentRewriter | PASS | Autoloaded in lib/source_monitor.rb |
+| Autoloading | Images::Downloader | PASS | Autoloaded in lib/source_monitor.rb |
+
+## Anti-Pattern Scan
+
+| Pattern | Found | Location | Severity |
+|---------|-------|----------|----------|
+| TODO/HACK/FIXME/XXX | NO | lib/source_monitor/images/* | N/A |
+| TODO/HACK/FIXME/XXX | NO | app/jobs/source_monitor/download_content_images_job.rb | N/A |
+| .html_safe | NO | lib/source_monitor/images/* | N/A |
+| Unsafe HTML rendering | NO | All new files | N/A |
+
+## Requirement Mapping
+
+| Requirement | Plan Ref | Artifact Evidence | Status |
+|-------------|----------|-------------------|--------|
+| REQ-24: Configurable image download option | PLAN-01 | ImagesSettings with download_to_active_storage (default false) | PASS |
+| REQ-24: Image detection in content | PLAN-01 | ContentRewriter#image_urls extracts URLs from HTML | PASS |
+| REQ-24: Active Storage attachment | PLAN-01 | ItemContent has_many_attached :images | PASS |
+| REQ-24: URL rewriting in content | PLAN-01 | ContentRewriter#rewrite replaces src attributes | PASS |
+| REQ-24: Image download service | PLAN-02 | Downloader downloads, validates size/content-type | PASS |
+| REQ-24: Background job orchestration | PLAN-02 | DownloadContentImagesJob performs full pipeline | PASS |
+| REQ-24: Pipeline integration | PLAN-02 | EntryProcessor enqueues job for new items | PASS |
+| REQ-24: Graceful fallback on failures | PLAN-02 | Failed downloads preserve original URLs | PASS |
+| REQ-24: Zero behavior change when disabled | PLAN-02 | Default config (false) = no jobs enqueued | PASS |
+| REQ-24: Documentation | PLAN-02 | sm-configure skill updated with config.images | PASS |
+
+## Summary
+
+**Tier:** standard
+
+**Result:** PASS
+
+**Passed:** 34/34
+
+**Failed:** None
+
+### Highlights
+
+- **Configuration layer:** ImagesSettings with 4 configurable attributes (download_to_active_storage defaults to false, max_download_size 10MB, download_timeout 30s, allowed_content_types 5 image MIME types). Fully integrated into SourceMonitor.config.images.
+
+- **Model layer:** ItemContent has_many_attached :images via Active Storage. Active Storage tables installed in dummy app for testing.
+
+- **HTML processing:** ContentRewriter uses Nokolexbor to extract image URLs and rewrite img src attributes. Handles relative URLs, data: URIs, malformed URLs gracefully.
+
+- **Download service:** Downloader validates content type and size, derives filenames, returns nil on any failure for graceful fallback.
+
+- **Background job:** DownloadContentImagesJob orchestrates the full pipeline: extract URLs, download images, attach to ItemContent, rewrite item.content with Active Storage blob paths. Idempotent (skips if images already attached). Runs on fetch queue.
+
+- **Integration hook:** EntryProcessor calls enqueue_image_download after item creation. Only fires when config enabled and item has HTML content. Wrapped in rescue to prevent feed processing breakage.
+
+- **Documentation:** sm-configure skill and reference updated with full config.images section.
+
+- **Test coverage:** 74 new tests (15 + 27 + 6 + 11 + 10 + 5) covering all scenarios. Full suite: 967 runs, 3100 assertions, 0 failures.
+
+- **Code quality:** RuboCop 389 files, 0 offenses. All files have frozen_string_literal. No anti-patterns detected.
+
+- **Zero behavior change:** Default config (download_to_active_storage = false) means no jobs enqueued, no Active Storage calls, no content rewriting. Existing pipelines unaffected.
+
+### Deviations
+
+None. Both plans executed as specified. All must_haves verified. All artifacts present and tested.
+
+### Next Steps
+
+Phase 5 complete. REQ-24 fully satisfied. Engine now supports configurable image downloads to Active Storage with graceful fallback on all failure modes.

data/.vbw-planning/phases/05-active-storage-images/PLAN-01-SUMMARY.md

@@ -0,0 +1,69 @@
+# PLAN-01 Summary: images-config-model-rewriter
+
+## Status: COMPLETE
+
+## What Was Built
+
+### Task 1: Create ImagesSettings config class
+- Added `ImagesSettings` class with `download_to_active_storage` (default false), `max_download_size` (10MB), `download_timeout` (30s), `allowed_content_types` (5 image MIME types)
+- Wired into `Configuration` as `config.images` attr_reader, initialized in constructor
+- Added `require` to configuration.rb and `Images` module autoload to lib/source_monitor.rb
+- 15 tests covering defaults, accessors, reset!, download_enabled?, integration with SourceMonitor.config
+
+### Task 2: Install Active Storage and add attachment
+- Created Active Storage migration in dummy app (`test/dummy/db/migrate/20260212000000_create_active_storage_tables.rb`)
+- Active Storage tables (blobs, attachments, variant_records) added to dummy app schema
+- Added `has_many_attached :images` to `ItemContent` model
+- Created 1x1 PNG test fixture at `test/fixtures/files/test_image.png`
+- 6 tests covering belongs_to, validates presence, images attachment, empty collection, single/multiple attach
+
+### Task 3: Create ContentRewriter class
+- Added `ContentRewriter` class in `lib/source_monitor/images/content_rewriter.rb` using Nokolexbor
+- `image_urls` method extracts absolute image URLs from `<img>` tags, skipping data: URIs and invalid URLs
+- `rewrite` method yields each downloadable URL to a block and replaces src with return value; preserves original on nil return
+- Handles relative URL resolution via `base_url` parameter, whitespace in src, self-closing tags
+- 27 tests covering extraction, rewriting, and edge cases
+
+### Task 4: Update existing configuration tests
+- Added 4 images settings tests to `configuration_test.rb` (accessible, defaults, configure block, reset)
+- Added `ImagesSettingsInSettingsTest` class to `settings_test.rb` with 4 tests (defaults, reset, download_enabled?, type check)
+
+### Task 5: Full verification
+- All 941 tests pass (0 failures, 0 errors)
+- RuboCop: 384 files, 0 offenses (12 array bracket spacing issues auto-fixed)
+
+## Files Modified
+- `lib/source_monitor/configuration/images_settings.rb` (new -- ImagesSettings class)
+- `lib/source_monitor/configuration.rb` (added images_settings require, attr_reader, initialization)
+- `lib/source_monitor.rb` (added Images module autoload)
+- `app/models/source_monitor/item_content.rb` (added has_many_attached :images)
+- `lib/source_monitor/images/content_rewriter.rb` (new -- ContentRewriter class)
+- `test/lib/source_monitor/configuration/images_settings_test.rb` (new -- 15 tests)
+- `test/models/source_monitor/item_content_test.rb` (new -- 6 tests)
+- `test/lib/source_monitor/images/content_rewriter_test.rb` (new -- 27 tests)
+- `test/lib/source_monitor/configuration_test.rb` (added 4 images tests)
+- `test/lib/source_monitor/configuration/settings_test.rb` (added 4 images tests)
+- `test/dummy/db/migrate/20260212000000_create_active_storage_tables.rb` (new -- Active Storage migration)
+- `test/dummy/db/schema.rb` (updated with Active Storage tables)
+- `test/fixtures/files/test_image.png` (new -- 1x1 PNG fixture)
+
+## Commits
+- `884a3f6` feat(05-01): create-images-settings
+- `52f4291` feat(05-01): install-active-storage-and-add-attachment
+- `e199999` feat(05-01): create-content-rewriter
+- `3954cb1` test(05-01): integration-test-and-config-test-update
+- `05705a8` style(05-01): fix array bracket spacing in content_rewriter_test
+
+## Requirements Satisfied
+- REQ-24 config: ImagesSettings with download_to_active_storage toggle (default false), size/timeout/content-type limits
+- REQ-24 attachment: ItemContent has has_many_attached :images via Active Storage
+- REQ-24 detection: ContentRewriter.image_urls extracts downloadable image URLs from HTML content
+- REQ-24 URL replacement: ContentRewriter.rewrite replaces img src attributes via block
+- REQ-24 graceful fallback: ContentRewriter preserves original URLs when rewrite block returns nil
+
+## Verification Results
+- `bin/rails test`: 941 runs, 3045 assertions, 0 failures, 0 errors
+- `bin/rubocop`: 384 files inspected, 0 offenses
+
+## Deviations
+None. All tasks executed as specified in the plan.