source_monitor 0.6.0 → 0.7.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (35) hide show
  1. checksums.yaml +4 -4
  2. data/.claude/commands/release.md +45 -22
  3. data/.gitignore +7 -0
  4. data/.vbw-planning/ROADMAP.md +53 -0
  5. data/.vbw-planning/STATE.md +27 -0
  6. data/.vbw-planning/phases/01-aia-certificate-resolution/.context-dev.md +17 -0
  7. data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-01-SUMMARY.md +26 -0
  8. data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-01.md +71 -0
  9. data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-02-SUMMARY.md +16 -0
  10. data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-02.md +56 -0
  11. data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-03-SUMMARY.md +17 -0
  12. data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-03.md +98 -0
  13. data/.vbw-planning/phases/02-test-performance/.context-dev.md +75 -0
  14. data/.vbw-planning/phases/02-test-performance/.context-lead.md +89 -0
  15. data/.vbw-planning/phases/02-test-performance/.context-qa.md +23 -0
  16. data/.vbw-planning/phases/02-test-performance/02-RESEARCH.md +56 -0
  17. data/.vbw-planning/phases/02-test-performance/02-VERIFICATION.md +51 -0
  18. data/.vbw-planning/phases/02-test-performance/PLAN-01-SUMMARY.md +37 -0
  19. data/.vbw-planning/phases/02-test-performance/PLAN-01.md +156 -0
  20. data/.vbw-planning/phases/02-test-performance/PLAN-02-SUMMARY.md +33 -0
  21. data/.vbw-planning/phases/02-test-performance/PLAN-02.md +120 -0
  22. data/.vbw-planning/phases/02-test-performance/PLAN-03-SUMMARY.md +30 -0
  23. data/.vbw-planning/phases/02-test-performance/PLAN-03.md +154 -0
  24. data/.vbw-planning/phases/02-test-performance/PLAN-04-SUMMARY.md +28 -0
  25. data/.vbw-planning/phases/02-test-performance/PLAN-04.md +133 -0
  26. data/CHANGELOG.md +35 -0
  27. data/Gemfile.lock +1 -1
  28. data/VERSION +1 -1
  29. data/lib/source_monitor/fetching/feed_fetcher/entry_processor.rb +5 -0
  30. data/lib/source_monitor/fetching/feed_fetcher/source_updater.rb +7 -4
  31. data/lib/source_monitor/fetching/feed_fetcher.rb +49 -3
  32. data/lib/source_monitor/items/item_creator.rb +31 -5
  33. data/lib/source_monitor/version.rb +1 -1
  34. data/lib/tasks/test_fast.rake +11 -0
  35. metadata +24 -1
@@ -0,0 +1,89 @@
1
+ ## Phase 02 Context (Compiled)
2
+
3
+ ### Goal
4
+ Not available
5
+
6
+ ### Success Criteria
7
+ Not available
8
+
9
+ ### Requirements (Not available)
10
+ No matching requirements found
11
+
12
+
13
+ ### Active Decisions
14
+
15
+ | Decision | Date | Context |
16
+ |----------|------|---------|
17
+ | Single-phase milestone for AIA fix | 2026-02-17 | Complete plan already validated; no scoping needed |
18
+ | 3 plans with wave parallelism | 2026-02-17 | Plans 01+02 (wave 1, disjoint files), Plan 03 (wave 2, integration) |
19
+
20
+ ### Codebase Map Available
21
+ Codebase mapping exists in `.vbw-planning/codebase/`. Key files:
22
+ - `ARCHITECTURE.md`
23
+ - `CONCERNS.md`
24
+ - `PATTERNS.md`
25
+ - `DEPENDENCIES.md`
26
+ - `STRUCTURE.md`
27
+ - `CONVENTIONS.md`
28
+ - `TESTING.md`
29
+ - `STACK.md`
30
+
31
+ Read ARCHITECTURE.md, CONCERNS.md, and STRUCTURE.md first to bootstrap codebase understanding.
32
+
33
+ ### Research Findings
34
+ # Phase 2: Test Performance - Research
35
+
36
+ ## Investigation Method
37
+ 3-agent competing hypothesis team (H1: DB/parallelism, H2: HTTP/VCR, H3: test helpers)
38
+
39
+ ## Findings
40
+
41
+ ### Time Budget (133s total, 1031 tests)
42
+ | Component | Time | % | Root Cause |
43
+ |-----------|------|---|------------|
44
+ | FeedFetcherTest (71 tests) | 84.8s | 64% | Monolithic class, CPU-bound (Feedjira parse, SHA256, content extraction) |
45
+ | Integration tests (9 tests) | 30.8s | 23% | Subprocess spawning (bundle exec rails g) |
46
+ | System tests (22 tests) | ~5s | 4% | Selenium/Chrome |
47
+ | Debug log IO | 5-15s | 8% | 95MB :debug output to disk |
48
+ | Everything else (~930 tests) | ~15s | 11% | Fast |
49
+
50
+ ### Why Parallelism Doesn't Help Today
51
+ - Minitest distributes by CLASS, not by test
52
+ - FeedFetcherTest = 1 class with 71 tests = 1 worker gets all 84.8s
53
+ - With 8 workers: max(84.8, 25.6, ~3) ≈ 85s + overhead
54
+
55
+ ### DB Is NOT the Bottleneck
56
+ - SQL: 3.639s of 69.4s (5.25%) via EventProf
57
+ - schema.rb used (fast load)
58
+ - Transactional tests enabled (proper rollback)
59
+
60
+ ### HTTP Mocking Is NOT the Bottleneck
61
+ - Only 4 VCR cassettes (456KB total), 83 WebMock stubs across 9 files
62
+ - WebMock configured once globally, auto-resets
63
+ - Only finding: default_cert_store recreated per connection (~0.5s total)
64
+
65
+ ### Test Helpers Are Cheap
66
+ - reset_configuration!: microseconds per call (pure Ruby)
67
+ - create_source!: 1 INSERT per call, 158 calls total
68
+ - with_inline_jobs: 4 calls total, negligible
69
+
70
+ ### Profiling Data (tmp/test_prof/)
71
+ - TagProf: lib tests 38.0s (55%), integration 28.5s (41%), controllers 0.97s, jobs 0.53s, models 0.47s
72
+ - EventProf: FeedFetcherTest 0.888s SQL in 34.3s total (2.6% SQL)
73
+
74
+ ## Relevant Patterns
75
+ - TestProf before_all available but used in only 1 of 54 eligible files
76
+ - Thread-based parallelism already proven in coverage mode
77
+ - PG fork segfault only on single-file runs, NOT full suite
78
+
79
+ ## Risks
80
+ - Thread safety: reset_configuration! may need thread-local config or mutex
81
+ - Test isolation: splitting FeedFetcherTest must preserve test independence
82
+ - before_all: only safe for tests that don't mutate shared data
83
+
84
+ ## Recommendations (by impact)
85
+ 1. Split FeedFetcherTest into 5-6 classes → enables parallelism, -50s
86
+ 2. Set config.log_level = :warn → -5-15s
87
+ 3. Tag/skip integration tests in dev → -30s dev experience
88
+ 4. Switch to thread-based parallelism → enables splitting benefit
89
+ 5. Adopt before_all in DB-heavy test files → -3-5s
@@ -0,0 +1,23 @@
1
+ ## Phase 02 Verification Context
2
+
3
+ ### Goal
4
+ Not available
5
+
6
+ ### Success Criteria
7
+ Not available
8
+
9
+ ### Requirements to Verify
10
+ No matching requirements found
11
+
12
+ ### Codebase Map Available
13
+ Codebase mapping exists in `.vbw-planning/codebase/`. Key files:
14
+ - `ARCHITECTURE.md`
15
+ - `CONCERNS.md`
16
+ - `PATTERNS.md`
17
+ - `DEPENDENCIES.md`
18
+ - `STRUCTURE.md`
19
+ - `CONVENTIONS.md`
20
+ - `TESTING.md`
21
+ - `STACK.md`
22
+
23
+ Read TESTING.md, CONCERNS.md, and ARCHITECTURE.md first to bootstrap codebase understanding.
@@ -0,0 +1,56 @@
1
+ # Phase 2: Test Performance - Research
2
+
3
+ ## Investigation Method
4
+ 3-agent competing hypothesis team (H1: DB/parallelism, H2: HTTP/VCR, H3: test helpers)
5
+
6
+ ## Findings
7
+
8
+ ### Time Budget (133s total, 1031 tests)
9
+ | Component | Time | % | Root Cause |
10
+ |-----------|------|---|------------|
11
+ | FeedFetcherTest (71 tests) | 84.8s | 64% | Monolithic class, CPU-bound (Feedjira parse, SHA256, content extraction) |
12
+ | Integration tests (9 tests) | 30.8s | 23% | Subprocess spawning (bundle exec rails g) |
13
+ | System tests (22 tests) | ~5s | 4% | Selenium/Chrome |
14
+ | Debug log IO | 5-15s | 8% | 95MB :debug output to disk |
15
+ | Everything else (~930 tests) | ~15s | 11% | Fast |
16
+
17
+ ### Why Parallelism Doesn't Help Today
18
+ - Minitest distributes by CLASS, not by test
19
+ - FeedFetcherTest = 1 class with 71 tests = 1 worker gets all 84.8s
20
+ - With 8 workers: max(84.8, 25.6, ~3) ≈ 85s + overhead
21
+
22
+ ### DB Is NOT the Bottleneck
23
+ - SQL: 3.639s of 69.4s (5.25%) via EventProf
24
+ - schema.rb used (fast load)
25
+ - Transactional tests enabled (proper rollback)
26
+
27
+ ### HTTP Mocking Is NOT the Bottleneck
28
+ - Only 4 VCR cassettes (456KB total), 83 WebMock stubs across 9 files
29
+ - WebMock configured once globally, auto-resets
30
+ - Only finding: default_cert_store recreated per connection (~0.5s total)
31
+
32
+ ### Test Helpers Are Cheap
33
+ - reset_configuration!: microseconds per call (pure Ruby)
34
+ - create_source!: 1 INSERT per call, 158 calls total
35
+ - with_inline_jobs: 4 calls total, negligible
36
+
37
+ ### Profiling Data (tmp/test_prof/)
38
+ - TagProf: lib tests 38.0s (55%), integration 28.5s (41%), controllers 0.97s, jobs 0.53s, models 0.47s
39
+ - EventProf: FeedFetcherTest 0.888s SQL in 34.3s total (2.6% SQL)
40
+
41
+ ## Relevant Patterns
42
+ - TestProf before_all available but used in only 1 of 54 eligible files
43
+ - Thread-based parallelism already proven in coverage mode
44
+ - PG fork segfault only on single-file runs, NOT full suite
45
+
46
+ ## Risks
47
+ - Thread safety: reset_configuration! may need thread-local config or mutex
48
+ - Test isolation: splitting FeedFetcherTest must preserve test independence
49
+ - before_all: only safe for tests that don't mutate shared data
50
+
51
+ ## Recommendations (by impact)
52
+ 1. Split FeedFetcherTest into 5-6 classes → enables parallelism, -50s
53
+ 2. Set config.log_level = :warn → -5-15s
54
+ 3. Tag/skip integration tests in dev → -30s dev experience
55
+ 4. Switch to thread-based parallelism → enables splitting benefit
56
+ 5. Adopt before_all in DB-heavy test files → -3-5s
@@ -0,0 +1,51 @@
1
+ # Phase 02 Verification: Test Performance
2
+
3
+ **Verdict: PASS**
4
+ **Date:** 2026-02-18
5
+ **Tier:** Deep
6
+
7
+ ## Requirements Verification
8
+
9
+ | Requirement | Status | Evidence |
10
+ |-------------|--------|----------|
11
+ | REQ-PERF-01: FeedFetcherTest split into 5+ classes | PASS | 6 files created in test/lib/source_monitor/fetching/ (success, error_handling, adaptive_interval, retry_circuit, entry_processing, utilities) |
12
+ | REQ-PERF-02: Test log level set to :warn | PASS | config.log_level = :warn in test/dummy/config/environments/test.rb |
13
+ | REQ-PERF-03: Integration tests excludable | PASS | lib/tasks/test_fast.rake provides test:fast task (1022 tests vs 1033 full) |
14
+ | REQ-PERF-04: Default parallelism switched to threads | PASS | parallelize(workers: worker_count, with: :threads) in test_helper.rb |
15
+ | REQ-PERF-05: DB-heavy files use setup_once/before_all | PASS | 5 files now use setup_once (up from 1) |
16
+
17
+ ## Checks Performed
18
+
19
+ ### Structural Checks
20
+ - [x] 6 split test files exist in test/lib/source_monitor/fetching/
21
+ - [x] Original feed_fetcher_test.rb deleted
22
+ - [x] Shared helper module exists (feed_fetcher_test_helper.rb)
23
+ - [x] Each split file includes FeedFetcherTestHelper
24
+ - [x] Test count preserved: 71 tests across 6 files (5+13+6+6+8+33)
25
+ - [x] config.log_level = :warn present in test.rb
26
+ - [x] lib/tasks/test_fast.rake exists and is valid Ruby
27
+ - [x] parallelize uses with: :threads for all modes
28
+ - [x] setup_once used in 5 files (sources_index_metrics, source_activity_rates, source_fetch_interval_distribution, upcoming_fetch_schedule, query_test)
29
+
30
+ ### Runtime Checks
31
+ - [x] Full test suite: 1033 tests, 0 failures, 0 errors
32
+ - [x] Single-file runs work without PARALLEL_WORKERS=1 (PG segfault eliminated)
33
+ - [x] Each split file passes individually with PARALLEL_WORKERS=1
34
+ - [x] test:fast runs 1022 tests (excludes 11 integration/system tests)
35
+ - [x] Two consecutive full suite runs: 0 flaky failures
36
+
37
+ ### Quality Checks
38
+ - [x] RuboCop: 0 offenses across all modified files
39
+ - [x] Brakeman: 0 warnings
40
+ - [x] No test isolation regressions
41
+
42
+ ### Commits
43
+ - 912665f: perf(02-03): adopt setup_once/before_all in DB-heavy test files
44
+ - a951c95: feat(02-01): split FeedFetcherTest into 6 concern-based test classes
45
+ - edbfe23: perf(02-02): reduce test log IO and add test:fast rake task
46
+ - eceb06d: perf(test): switch default parallelism from forks to threads
47
+
48
+ ## Notes
49
+
50
+ - TestProf emits cosmetic warning "before_all is not implemented for parallalization with threads" — does not affect correctness. before_all/setup_once works correctly because single-file runs stay below parallelization threshold, and full suite distributes by class.
51
+ - PLAN-02 deviated from plan: --exclude-pattern is not a Minitest feature, so Dir glob approach was used instead. Functionally equivalent.
@@ -0,0 +1,37 @@
1
+ ---
2
+ status: complete
3
+ phase: "02"
4
+ plan: "01"
5
+ title: "Split FeedFetcherTest into Concern-Based Classes"
6
+ commits:
7
+ - hash: a951c95
8
+ message: "feat(02-01): split FeedFetcherTest into 6 concern-based test classes"
9
+ tasks_completed: 4
10
+ tasks_total: 4
11
+ deviations: []
12
+ ---
13
+
14
+ ## What Was Built
15
+
16
+ Split the monolithic `FeedFetcherTest` (71 tests, 1350 lines, single class) into 6 independent test classes plus a shared helper module. Each file is independently runnable with `PARALLEL_WORKERS=1`. All 1033 tests pass, RuboCop zero offenses.
17
+
18
+ Test distribution:
19
+ - `FeedFetcherSuccessTest`: 5 tests (RSS/Atom/JSON fetching, ETag/304, Netflix cassette)
20
+ - `FeedFetcherErrorHandlingTest`: 13 tests (Faraday error wrapping, AIA resolution, failure recording)
21
+ - `FeedFetcherAdaptiveIntervalTest`: 6 tests (interval increase/decrease, min/max bounds, disabled mode)
22
+ - `FeedFetcherRetryCircuitTest`: 6 tests (retry state, circuit breaker, policy error handling)
23
+ - `FeedFetcherEntryProcessingTest`: 8 tests (entry creation/update tracking, error normalization, digest fallbacks)
24
+ - `FeedFetcherUtilitiesTest`: 33 tests (headers, jitter, metadata, signature, config helpers, parsing)
25
+
26
+ ## Files Modified
27
+
28
+ | Action | Path |
29
+ |--------|------|
30
+ | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_test_helper.rb` |
31
+ | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_success_test.rb` |
32
+ | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_error_handling_test.rb` |
33
+ | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_adaptive_interval_test.rb` |
34
+ | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_retry_circuit_test.rb` |
35
+ | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_entry_processing_test.rb` |
36
+ | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_utilities_test.rb` |
37
+ | DELETE | `test/lib/source_monitor/fetching/feed_fetcher_test.rb` |
@@ -0,0 +1,156 @@
1
+ ---
2
+ phase: "02"
3
+ plan: "01"
4
+ title: "Split FeedFetcherTest into Concern-Based Classes"
5
+ wave: 1
6
+ depends_on: []
7
+ must_haves:
8
+ - "REQ-PERF-01: FeedFetcherTest split into 6+ independent test files by concern"
9
+ - "Original feed_fetcher_test.rb deleted or replaced with require-only shim"
10
+ - "Each new test file is independently runnable with PARALLEL_WORKERS=1"
11
+ - "All 71 FeedFetcherTest tests pass individually and in full suite"
12
+ - "Shared build_source and body_digest helpers extracted to shared module"
13
+ - "RuboCop zero offenses on all new/modified test files"
14
+ skills_used: []
15
+ ---
16
+
17
+ # Plan 01: Split FeedFetcherTest into Concern-Based Classes
18
+
19
+ ## Objective
20
+
21
+ Split the monolithic `FeedFetcherTest` (71 tests, 84.8s, 64% of total runtime) into 6+ smaller test classes by concern. This is the single highest-impact optimization: Minitest parallelizes by class, so one 71-test class gets assigned to one worker. Splitting enables parallel distribution across all CPU cores.
22
+
23
+ ## Context
24
+
25
+ - `@` `test/lib/source_monitor/fetching/feed_fetcher_test.rb` -- 1350-line monolithic test class (the file to split)
26
+ - `@` `test/test_helper.rb` -- base test setup with `create_source!` and `clean_source_monitor_tables!`
27
+ - `@` `test/test_prof.rb` -- TestProf `before_all` and `setup_once` support
28
+
29
+ **Rationale:** The test file already has section comments (Task 1-6) that map to concern groups. The `build_source` and `body_digest` private helpers are shared across all tests and must be extracted to a module.
30
+
31
+ ## Tasks
32
+
33
+ ### Task 1: Create shared helper module for FeedFetcher tests
34
+
35
+ Create `test/lib/source_monitor/fetching/feed_fetcher_test_helper.rb`:
36
+
37
+ ```ruby
38
+ # frozen_string_literal: true
39
+
40
+ module SourceMonitor
41
+ module Fetching
42
+ module FeedFetcherTestHelper
43
+ private
44
+
45
+ def build_source(name:, feed_url:, fetch_interval_minutes: 360, adaptive_fetching_enabled: true)
46
+ create_source!(
47
+ name: name,
48
+ feed_url: feed_url,
49
+ fetch_interval_minutes: fetch_interval_minutes,
50
+ adaptive_fetching_enabled: adaptive_fetching_enabled
51
+ )
52
+ end
53
+
54
+ def body_digest(body)
55
+ Digest::SHA256.hexdigest(body)
56
+ end
57
+ end
58
+ end
59
+ end
60
+ ```
61
+
62
+ ### Task 2: Create the 6 split test files
63
+
64
+ Split the 71 tests into these files, each requiring `test_helper` and `feed_fetcher_test_helper`:
65
+
66
+ 1. **`feed_fetcher_success_test.rb`** (~13 tests) -- Success paths: RSS/Atom/JSON fetching, log entries, instrumentation notifications, ETag/304 handling, Netflix cassette test. Tests: "continues processing when an item creation fails", "fetches an RSS feed and records log entries", "reuses etag and handles 304", "parses rss atom and json feeds via feedjira", "fetches Netflix Tech Blog feed via Medium RSS".
67
+
68
+ 2. **`feed_fetcher_error_handling_test.rb`** (~12 tests) -- Error wrapping and connection failures: all Faraday error type wrapping, AIA certificate resolution tests (retry on SSL, nil resolve, non-SSL skip), generic Faraday::Error, unexpected StandardError, HTTPError from ClientError, re-raise without double-wrap. Tests from "Task 2" section plus AIA tests.
69
+
70
+ 3. **`feed_fetcher_adaptive_interval_test.rb`** (~8 tests) -- Adaptive fetch interval: decrease on content change, increase on no change, configured settings, min/max bounds, failure increase with backoff, disabled adaptive fetching. Tests from the interval section.
71
+
72
+ 4. **`feed_fetcher_retry_circuit_test.rb`** (~7 tests) -- Retry strategy and circuit breaker: reset on success, reset on 304, apply retry state, circuit open when exhausted, next_fetch_at earliest logic, policy error handling. Tests from "Task 1" section.
73
+
74
+ 5. **`feed_fetcher_entry_processing_test.rb`** (~7 tests) -- Entry processing: empty entries, error normalization (with guid, without guid), created/updated tracking, unchanged items, entries_digest fallback (url, title), failure result empty processing. Tests from "Task 4" section plus "process_feed_entries tracks created and updated" and "unchanged items".
75
+
76
+ 6. **`feed_fetcher_utilities_test.rb`** (~16 tests) -- Utility methods: jitter_offset, adjusted_interval_with_jitter, body_digest, updated_metadata, feed_signature_changed?, configured_seconds, configured_positive, configured_non_negative, interval_minutes_for, parse_http_time, extract_numeric. Tests from "Task 5" section plus "Task 3" header tests (If-Modified-Since, custom_headers, ETag update, Last-Modified update/304/unparseable).
77
+
78
+ Each file follows this pattern:
79
+ ```ruby
80
+ # frozen_string_literal: true
81
+
82
+ require "test_helper"
83
+ require "faraday"
84
+ require "uri"
85
+ require "digest"
86
+ require_relative "feed_fetcher_test_helper"
87
+
88
+ module SourceMonitor
89
+ module Fetching
90
+ class FeedFetcherSuccessTest < ActiveSupport::TestCase
91
+ include FeedFetcherTestHelper
92
+ # tests here...
93
+ end
94
+ end
95
+ end
96
+ ```
97
+
98
+ ### Task 3: Delete original feed_fetcher_test.rb
99
+
100
+ After all 6 new files are created and verified, delete the original `test/lib/source_monitor/fetching/feed_fetcher_test.rb`. Do NOT keep a shim -- the new files are self-contained.
101
+
102
+ ### Task 4: Verify all tests pass and lint clean
103
+
104
+ Run verification:
105
+ ```bash
106
+ # Run each new file individually
107
+ PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/fetching/feed_fetcher_success_test.rb
108
+ PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/fetching/feed_fetcher_error_handling_test.rb
109
+ PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/fetching/feed_fetcher_adaptive_interval_test.rb
110
+ PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/fetching/feed_fetcher_retry_circuit_test.rb
111
+ PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/fetching/feed_fetcher_entry_processing_test.rb
112
+ PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/fetching/feed_fetcher_utilities_test.rb
113
+
114
+ # Full suite
115
+ bin/rails test
116
+
117
+ # Lint
118
+ bin/rubocop test/lib/source_monitor/fetching/
119
+ ```
120
+
121
+ Ensure the total test count remains 1031+ (no tests lost or duplicated).
122
+
123
+ ## Files
124
+
125
+ | Action | Path |
126
+ |--------|------|
127
+ | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_test_helper.rb` |
128
+ | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_success_test.rb` |
129
+ | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_error_handling_test.rb` |
130
+ | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_adaptive_interval_test.rb` |
131
+ | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_retry_circuit_test.rb` |
132
+ | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_entry_processing_test.rb` |
133
+ | CREATE | `test/lib/source_monitor/fetching/feed_fetcher_utilities_test.rb` |
134
+ | DELETE | `test/lib/source_monitor/fetching/feed_fetcher_test.rb` |
135
+
136
+ ## Verification
137
+
138
+ ```bash
139
+ # Individual file runs (PARALLEL_WORKERS=1 due to PG fork segfault on single files)
140
+ PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/fetching/feed_fetcher_success_test.rb
141
+ PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/fetching/feed_fetcher_utilities_test.rb
142
+
143
+ # Full suite (all 1031+ tests pass)
144
+ bin/rails test
145
+
146
+ # Lint
147
+ bin/rubocop test/lib/source_monitor/fetching/
148
+ ```
149
+
150
+ ## Success Criteria
151
+
152
+ - 6+ new test files exist in `test/lib/source_monitor/fetching/`
153
+ - Original `feed_fetcher_test.rb` deleted
154
+ - `grep -c "class Feed" test/lib/source_monitor/fetching/*_test.rb` shows 6+ classes
155
+ - All 1031+ tests pass in full suite
156
+ - Each file runnable independently with PARALLEL_WORKERS=1
@@ -0,0 +1,33 @@
1
+ ---
2
+ plan: "02"
3
+ phase: "02"
4
+ title: "Log Level Reduction and Integration Test Tagging"
5
+ status: complete
6
+ commits:
7
+ - hash: edbfe23
8
+ message: "perf(02-02): reduce test log IO and add test:fast rake task"
9
+ tasks_completed: 4
10
+ tasks_total: 4
11
+ files_modified:
12
+ - test/dummy/config/environments/test.rb
13
+ files_created:
14
+ - lib/tasks/test_fast.rake
15
+ ---
16
+
17
+ ## What Was Built
18
+
19
+ - Set `config.log_level = :warn` in test environment to eliminate ~95MB of debug log IO per test run
20
+ - Created `lib/tasks/test_fast.rake` providing `test:fast` rake task that excludes integration/ and system/ directories
21
+ - Verified all 4 integration test files already in `test/integration/` (no moves needed)
22
+ - Full suite: 1033 runs, 0 failures; Fast mode: 1022 runs, 0 failures
23
+
24
+ ## Files Modified
25
+
26
+ - `test/dummy/config/environments/test.rb` — added `config.log_level = :warn` after `config.cache_store = :null_store`
27
+ - `lib/tasks/test_fast.rake` — new rake task `test:fast` using Dir glob to exclude integration and system test files
28
+
29
+ ## Deviations
30
+
31
+ - Plan specified `--exclude-pattern` flag for minitest but this flag does not exist in Rails/Minitest. Replaced with Dir glob approach that rejects `test/integration/` and `test/system/` paths (DEVN-01 Minor).
32
+ - Also excluded `test/system/` from fast mode since `bin/rails test` already excludes system tests by default — this makes `test:fast` equivalent to `bin/rails test` minus integration tests.
33
+ - Rake task accessible as `bundle exec rake app:test:fast` from engine root (engine prefixes with `app:`).
@@ -0,0 +1,120 @@
1
+ ---
2
+ phase: "02"
3
+ plan: "02"
4
+ title: "Log Level Reduction and Integration Test Tagging"
5
+ wave: 1
6
+ depends_on: []
7
+ must_haves:
8
+ - "REQ-PERF-02: config.log_level = :warn in test/dummy/config/environments/test.rb"
9
+ - "REQ-PERF-03: Integration tests tagged so --exclude-pattern can skip them"
10
+ - "host_install_flow_test.rb and release_packaging_test.rb moved under test/integration/"
11
+ - "bin/rails test --exclude-pattern='**/integration/**' excludes slow integration tests"
12
+ - "bin/rails test runs all tests including integration (default behavior preserved)"
13
+ - "RuboCop zero offenses on modified files"
14
+ skills_used: []
15
+ ---
16
+
17
+ # Plan 02: Log Level Reduction and Integration Test Tagging
18
+
19
+ ## Objective
20
+
21
+ Two quick wins that reduce test wall-clock time: (1) eliminate 95MB of debug log IO by setting test log level to `:warn` (saves 5-15s), and (2) ensure integration tests are organized under `test/integration/` so developers can exclude the 31s of subprocess-spawning tests during iterative development using `--exclude-pattern`.
22
+
23
+ ## Context
24
+
25
+ - `@` `test/dummy/config/environments/test.rb` -- currently has no explicit log_level (defaults to :debug)
26
+ - `@` `test/integration/host_install_flow_test.rb` -- slow integration test (subprocess spawning, ~15s)
27
+ - `@` `test/integration/release_packaging_test.rb` -- slow integration test (gem build + subprocess, ~15s)
28
+ - `@` `test/integration/engine_mounting_test.rb` -- fast integration test (route checks)
29
+ - `@` `test/integration/navigation_test.rb` -- empty placeholder test
30
+ - `@` `test/test_helper.rb` -- contains DEFAULT_TEST_EXCLUDE but does NOT need modification
31
+
32
+ **Rationale:** The research found 95MB of :debug log output during tests. Setting :warn eliminates this IO without losing any test coverage. Integration tests already live under `test/integration/` -- we just need to verify the exclude pattern works and document it.
33
+
34
+ ## Tasks
35
+
36
+ ### Task 1: Set test log level to :warn
37
+
38
+ In `test/dummy/config/environments/test.rb`, add after the `config.cache_store = :null_store` line:
39
+
40
+ ```ruby
41
+ # Reduce log IO in tests -- :debug generates ~95MB of output.
42
+ config.log_level = :warn
43
+ ```
44
+
45
+ This is a single-line addition. The research confirmed this saves 5-15s per run by eliminating disk IO for debug/info log messages.
46
+
47
+ ### Task 2: Verify integration test directory organization
48
+
49
+ Confirm all 4 integration test files are properly under `test/integration/`:
50
+ - `test/integration/host_install_flow_test.rb` (slow -- subprocess spawning)
51
+ - `test/integration/release_packaging_test.rb` (slow -- gem build)
52
+ - `test/integration/engine_mounting_test.rb` (fast -- route assertions)
53
+ - `test/integration/navigation_test.rb` (empty placeholder)
54
+
55
+ No file moves needed -- they are already in the correct location. The `--exclude-pattern` flag works with glob patterns on the file path.
56
+
57
+ ### Task 3: Add Rake task for fast test runs
58
+
59
+ Create `lib/tasks/test_fast.rake` with a convenience task:
60
+
61
+ ```ruby
62
+ # frozen_string_literal: true
63
+
64
+ namespace :test do
65
+ desc "Run tests excluding slow integration tests"
66
+ task fast: :environment do
67
+ $stdout.puts "Running tests excluding integration/ directory..."
68
+ system(
69
+ "bin/rails", "test",
70
+ "--exclude-pattern", "**/integration/**",
71
+ exception: true
72
+ )
73
+ end
74
+ end
75
+ ```
76
+
77
+ This provides `bin/rails test:fast` as a developer convenience that excludes the integration directory.
78
+
79
+ ### Task 4: Verify both test modes work
80
+
81
+ Run verification:
82
+ ```bash
83
+ # Full suite (all tests including integration)
84
+ bin/rails test
85
+
86
+ # Fast mode (excluding integration)
87
+ bin/rails test --exclude-pattern="**/integration/**"
88
+
89
+ # Lint modified files
90
+ bin/rubocop test/dummy/config/environments/test.rb lib/tasks/test_fast.rake
91
+ ```
92
+
93
+ Verify the fast mode excludes the integration tests and completes significantly faster.
94
+
95
+ ## Files
96
+
97
+ | Action | Path |
98
+ |--------|------|
99
+ | MODIFY | `test/dummy/config/environments/test.rb` |
100
+ | CREATE | `lib/tasks/test_fast.rake` |
101
+
102
+ ## Verification
103
+
104
+ ```bash
105
+ # Full suite passes
106
+ bin/rails test
107
+
108
+ # Exclude pattern works (fewer tests, no integration)
109
+ bin/rails test --exclude-pattern="**/integration/**"
110
+
111
+ # Lint
112
+ bin/rubocop test/dummy/config/environments/test.rb lib/tasks/test_fast.rake
113
+ ```
114
+
115
+ ## Success Criteria
116
+
117
+ - `grep "log_level" test/dummy/config/environments/test.rb` shows `:warn`
118
+ - `bin/rails test` runs all 1031+ tests (no regressions)
119
+ - `bin/rails test --exclude-pattern="**/integration/**"` runs successfully with fewer tests
120
+ - `lib/tasks/test_fast.rake` exists and is syntactically valid
@@ -0,0 +1,30 @@
1
+ ---
2
+ phase: 2
3
+ plan: 3
4
+ status: complete
5
+ ---
6
+ # Plan 03 Summary: Adopt before_all in DB-Heavy Test Files
7
+
8
+ ## Tasks Completed
9
+ - [x] Task 1: Convert sources_index_metrics_test.rb to setup_once (17 read-only tests)
10
+ - [x] Task 2: Convert 3 single-test files to setup_once for consistency
11
+ - [x] Task 3: Verify all converted files individually (PARALLEL_WORKERS=1)
12
+ - [x] Task 4: Full suite verification (1033 tests, 0 failures) and lint (0 offenses)
13
+
14
+ ## Commits
15
+ - 912665f: perf(02-03): adopt setup_once/before_all in DB-heavy test files
16
+
17
+ ## Files Modified
18
+ - test/lib/source_monitor/analytics/sources_index_metrics_test.rb (modified)
19
+ - test/lib/source_monitor/analytics/source_activity_rates_test.rb (modified)
20
+ - test/lib/source_monitor/analytics/source_fetch_interval_distribution_test.rb (modified)
21
+ - test/lib/source_monitor/dashboard/upcoming_fetch_schedule_test.rb (modified)
22
+
23
+ ## What Was Built
24
+ - Converted `sources_index_metrics_test.rb` from per-test setup to `setup_once` for shared fixture creation (3 sources + 3 items), following the reference pattern from `query_test.rb` (store IDs in setup_once, re-find records in per-test setup)
25
+ - Kept `travel_to`/`travel_back` in regular setup/teardown for thread-local time safety
26
+ - Converted 3 single-test files to `setup_once` for `clean_source_monitor_tables!` (functionally identical for single-test classes but normalizes the pattern)
27
+ - `setup_once` usage increased from 1 file to 5 files across the test suite
28
+
29
+ ## Deviations
30
+ - None