source_monitor 0.6.0 → 0.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.claude/commands/release.md +45 -22
- data/.gitignore +7 -0
- data/.vbw-planning/ROADMAP.md +53 -0
- data/.vbw-planning/STATE.md +27 -0
- data/.vbw-planning/phases/01-aia-certificate-resolution/.context-dev.md +17 -0
- data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-01-SUMMARY.md +26 -0
- data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-01.md +71 -0
- data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-02-SUMMARY.md +16 -0
- data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-02.md +56 -0
- data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-03-SUMMARY.md +17 -0
- data/.vbw-planning/phases/01-aia-certificate-resolution/PLAN-03.md +98 -0
- data/.vbw-planning/phases/02-test-performance/.context-dev.md +75 -0
- data/.vbw-planning/phases/02-test-performance/.context-lead.md +89 -0
- data/.vbw-planning/phases/02-test-performance/.context-qa.md +23 -0
- data/.vbw-planning/phases/02-test-performance/02-RESEARCH.md +56 -0
- data/.vbw-planning/phases/02-test-performance/02-VERIFICATION.md +51 -0
- data/.vbw-planning/phases/02-test-performance/PLAN-01-SUMMARY.md +37 -0
- data/.vbw-planning/phases/02-test-performance/PLAN-01.md +156 -0
- data/.vbw-planning/phases/02-test-performance/PLAN-02-SUMMARY.md +33 -0
- data/.vbw-planning/phases/02-test-performance/PLAN-02.md +120 -0
- data/.vbw-planning/phases/02-test-performance/PLAN-03-SUMMARY.md +30 -0
- data/.vbw-planning/phases/02-test-performance/PLAN-03.md +154 -0
- data/.vbw-planning/phases/02-test-performance/PLAN-04-SUMMARY.md +28 -0
- data/.vbw-planning/phases/02-test-performance/PLAN-04.md +133 -0
- data/CHANGELOG.md +35 -0
- data/Gemfile.lock +1 -1
- data/VERSION +1 -1
- data/lib/source_monitor/fetching/feed_fetcher/entry_processor.rb +5 -0
- data/lib/source_monitor/fetching/feed_fetcher/source_updater.rb +7 -4
- data/lib/source_monitor/fetching/feed_fetcher.rb +49 -3
- data/lib/source_monitor/items/item_creator.rb +31 -5
- data/lib/source_monitor/version.rb +1 -1
- data/lib/tasks/test_fast.rake +11 -0
- metadata +24 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 135499910675f0d424b88ec25e1503f3068fdbd78ffa553942438efd886b72bd
|
|
4
|
+
data.tar.gz: a6b03ef217569a206d7521f2d8b42f3c300ab6ddf41e144cabaf24535f92b8dc
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: b4e0ebe10a0211760f42d6a131cbf61eb09bf3f268006fbefe51636137c10ba0afc0413e2c39a46a8d3ff39037db07ba680a6f059599d7751b56a79d5deef1a3
|
|
7
|
+
data.tar.gz: 7bdfa9ca0995bb91757be78271ca171a14ec83cc54aef8d13b083446e3c23a96d4568718b40a944c83c802173124823609a36c405a68b8a0c81bb138891aa65b
|
data/.claude/commands/release.md
CHANGED
|
@@ -15,8 +15,10 @@ These are real issues encountered in previous releases. Each step below accounts
|
|
|
15
15
|
3. **VBW volatile files**: Files in `.vbw-planning/` (`.cost-ledger.json`, `.notification-log.jsonl`, `.session-log.jsonl`, `.hook-errors.log`) are continuously modified by VBW hooks. They should be in `.gitignore`. If they aren't, add them before proceeding.
|
|
16
16
|
4. **Pre-push hook**: The VBW pre-push hook at `.git/hooks/pre-push` requires the `VERSION` file to appear in the diff for any push. For new branches, it compares the commit against the working tree -- any dirty files will trigger a false positive. For force-pushes to existing branches where `VERSION` hasn't changed since the last push, use `--no-verify`.
|
|
17
17
|
5. **Single squashed commit**: Always create ONE commit on the release branch with ALL changes (version bump, changelog, Gemfile.lock, any fixes). Multiple commits cause pre-push hook issues.
|
|
18
|
-
6. **Diff coverage CI gate**: The `test` CI job enforces diff coverage. Any changed lines in source code (not just test files) must have test coverage.
|
|
19
|
-
7. **Local main divergence after merge**: After the PR merges,
|
|
18
|
+
6. **Diff coverage CI gate**: The `test` CI job enforces diff coverage. Any changed lines in source code (not just test files) must have test coverage. **This applies to ALL changes in the PR diff vs main, including unpushed commits made before the release started.** If the release includes source code changes (bug fixes, features), every changed source line must be covered.
|
|
19
|
+
7. **Local main divergence after merge**: After the PR merges, `gh pr merge --merge --delete-branch` will attempt to fast-forward local main. This usually succeeds automatically. If it doesn't (divergent history), you must `git reset --hard origin/main` to sync -- this requires user approval since the sandbox blocks it.
|
|
20
|
+
8. **Run local checks BEFORE pushing**: Always run `bin/rubocop` and `PARALLEL_WORKERS=1 bin/rails test` locally before the first push to the release branch. Each CI roundtrip (fail → fix → amend → force-push → re-run) costs ~5 minutes. In v0.7.0, skipping local checks caused two wasted CI cycles: first for uncovered diff lines, then for a RuboCop violation in the fix.
|
|
21
|
+
9. **Zsh glob nomatch**: Commands like `rm -f *.gem` fail in zsh when no files match. Always use `rm -f *.gem 2>/dev/null || true` or check existence first with `ls`.
|
|
20
22
|
|
|
21
23
|
## Step 1: Git Hygiene
|
|
22
24
|
|
|
@@ -112,7 +114,25 @@ The changelog follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) f
|
|
|
112
114
|
2. Verify the output shows the new version: `Using source_monitor X.Y.Z (was X.Y.Z-1)`.
|
|
113
115
|
3. If `bundle install` fails, resolve the issue before proceeding.
|
|
114
116
|
|
|
115
|
-
## Step 5:
|
|
117
|
+
## Step 5: Local Pre-flight Checks
|
|
118
|
+
|
|
119
|
+
**CRITICAL**: Run these checks BEFORE creating the release branch and pushing. Each CI failure → fix → amend → force-push cycle wastes ~5 minutes. In v0.7.0, skipping this step caused two wasted CI roundtrips.
|
|
120
|
+
|
|
121
|
+
1. **RuboCop**: Run `bin/rubocop` and fix any violations. Auto-fix with `bin/rubocop -a` if needed. This catches lint issues (like `SpaceInsideArrayLiteralBrackets`) that would fail the CI lint job.
|
|
122
|
+
|
|
123
|
+
2. **Tests**: Run `PARALLEL_WORKERS=1 bin/rails test` and ensure all tests pass.
|
|
124
|
+
|
|
125
|
+
3. **Diff coverage pre-check**: If the release includes source code changes beyond version/changelog/lockfile (check with `git diff --name-only origin/main`), review those files for uncovered branches. The CI diff coverage gate will reject any changed source lines without test coverage. Common blind spots:
|
|
126
|
+
- Fallback/else branches in new methods
|
|
127
|
+
- Error handling paths
|
|
128
|
+
- Guard clauses
|
|
129
|
+
If you find uncovered source lines, write tests for them NOW before creating the release commit — it's far cheaper than a CI roundtrip.
|
|
130
|
+
|
|
131
|
+
4. **Brakeman**: Run `bin/brakeman --no-pager` and ensure zero warnings.
|
|
132
|
+
|
|
133
|
+
Only proceed to Step 6 when all local checks pass.
|
|
134
|
+
|
|
135
|
+
## Step 6: Create Release Branch with Single Squashed Commit
|
|
116
136
|
|
|
117
137
|
**IMPORTANT**: All release changes MUST be in a single commit on the release branch. This avoids pre-push hook issues where individual commits are checked for VERSION changes.
|
|
118
138
|
|
|
@@ -124,13 +144,13 @@ The changelog follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) f
|
|
|
124
144
|
Also stage any other files that were changed (updated skills, docs, etc.).
|
|
125
145
|
3. Create a single commit:
|
|
126
146
|
```
|
|
127
|
-
chore: release vX.Y.Z
|
|
147
|
+
chore(release): release vX.Y.Z
|
|
128
148
|
```
|
|
129
149
|
4. Push the branch: `git push -u origin release/vX.Y.Z`
|
|
130
150
|
- If the pre-push hook blocks with a false positive (e.g., VBW files dirty in working tree despite being gitignored), use `git push -u --no-verify origin release/vX.Y.Z`. This is safe because we've verified VERSION is in the commit.
|
|
131
151
|
5. If the push fails for other reasons, diagnose and fix before proceeding.
|
|
132
152
|
|
|
133
|
-
## Step
|
|
153
|
+
## Step 7: Create PR
|
|
134
154
|
|
|
135
155
|
1. Create the PR using `gh pr create`:
|
|
136
156
|
- Title: `Release vX.Y.Z`
|
|
@@ -152,7 +172,7 @@ The changelog follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) f
|
|
|
152
172
|
- Base: `main`
|
|
153
173
|
2. Report the PR URL to the user.
|
|
154
174
|
|
|
155
|
-
## Step
|
|
175
|
+
## Step 8: Monitor CI Pipeline
|
|
156
176
|
|
|
157
177
|
Poll the CI status using repeated `gh pr checks <PR_NUMBER>` calls. The CI has 4 required jobs: `lint`, `security`, `test`, `release_verification` (release_verification only runs after test passes).
|
|
158
178
|
|
|
@@ -163,7 +183,7 @@ Poll the CI status using repeated `gh pr checks <PR_NUMBER>` calls. The CI has 4
|
|
|
163
183
|
|
|
164
184
|
### If CI PASSES (all checks green):
|
|
165
185
|
|
|
166
|
-
Continue to Step
|
|
186
|
+
Continue to Step 9. If Step 5 (local pre-flight) was done properly, CI should pass on the first attempt.
|
|
167
187
|
|
|
168
188
|
### If CI FAILS:
|
|
169
189
|
|
|
@@ -173,32 +193,32 @@ Continue to Step 8.
|
|
|
173
193
|
gh run view <RUN_ID> --log-failed | tail -80
|
|
174
194
|
```
|
|
175
195
|
3. **Common failure: diff coverage** -- If the `test` job fails on "Enforce diff coverage", it means changed source lines lack test coverage. Read the error to identify uncovered files/lines, write tests, and add them to the release commit.
|
|
176
|
-
4. **Common failure: Gemfile.lock frozen** -- If `bundle install` fails in CI with "frozen mode", you forgot to run `bundle install` locally (Step 4).
|
|
177
|
-
5.
|
|
178
|
-
|
|
196
|
+
4. **Common failure: Gemfile.lock frozen** -- If `bundle install` fails in CI with "frozen mode", you forgot to run `bundle install` locally (Step 4). Amend the commit with the updated lockfile.
|
|
197
|
+
5. **Common failure: RuboCop lint** -- If the `lint` job fails, a RuboCop violation slipped through. This should have been caught in Step 5.
|
|
198
|
+
6. **IMPORTANT: When fixing CI failures, run ALL local checks again before re-pushing.** Don't just fix the one failure — run `bin/rubocop` AND `PARALLEL_WORKERS=1 bin/rails test` to catch cascading issues. In v0.7.0, fixing a diff coverage failure introduced a RuboCop violation, requiring a third CI cycle.
|
|
199
|
+
7. Present failure details to the user and ask what to do:
|
|
200
|
+
- "Fix the issues and re-push" -- Fix issues, run ALL local checks (rubocop + tests), amend the commit (`git commit --amend --no-edit`), force push (`git push --force-with-lease --no-verify origin release/vX.Y.Z`), and restart CI monitoring.
|
|
179
201
|
- "Close the PR and abort" -- Close the PR, delete the branch, switch back to main.
|
|
180
202
|
- "Investigate manually" -- Stop and let the user handle it.
|
|
181
203
|
|
|
182
204
|
**Note on force pushes**: When force-pushing the release branch after amending, always use `--no-verify` because the pre-push hook will see the diff between old and new branch tips, and `VERSION` won't appear as changed (it's the same in both). This is expected and safe.
|
|
183
205
|
|
|
184
|
-
## Step
|
|
206
|
+
## Step 9: Auto-Merge PR
|
|
185
207
|
|
|
186
208
|
Once CI is green:
|
|
187
209
|
|
|
188
210
|
1. Merge the PR: `gh pr merge <PR_NUMBER> --merge --delete-branch`
|
|
211
|
+
- The `--delete-branch` flag also fetches and fast-forwards local main in most cases.
|
|
189
212
|
|
|
190
|
-
2. **Sync local main with remote
|
|
191
|
-
- The merge creates a merge commit on origin/main that doesn't exist locally.
|
|
192
|
-
- Local main may have different commits (pre-squash) than what was merged.
|
|
213
|
+
2. **Sync local main with remote**:
|
|
193
214
|
- Switch to main: `git checkout main`
|
|
194
|
-
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
- Verify with `git log --oneline -3` that local matches remote.
|
|
215
|
+
- The `gh pr merge` command usually auto-syncs local main via fast-forward. Verify with `git log --oneline -3` that local matches remote (`git rev-parse HEAD` == `git rev-parse origin/main`).
|
|
216
|
+
- If local is behind or diverged, try `git pull origin main`.
|
|
217
|
+
- If pull fails with conflicts or divergence (rare): ask the user to run `git reset --hard origin/main` (the sandbox blocks this command). Explain this is safe because the PR is merged and all changes are on origin/main.
|
|
198
218
|
|
|
199
219
|
3. Report: "PR #N merged successfully."
|
|
200
220
|
|
|
201
|
-
## Step
|
|
221
|
+
## Step 10: Tag the Release
|
|
202
222
|
|
|
203
223
|
1. Verify you're on main and synced with origin.
|
|
204
224
|
2. Create an annotated tag:
|
|
@@ -212,14 +232,17 @@ Once CI is green:
|
|
|
212
232
|
```
|
|
213
233
|
5. Report the release URL.
|
|
214
234
|
|
|
215
|
-
## Step
|
|
235
|
+
## Step 11: Build the Gem
|
|
216
236
|
|
|
217
|
-
1. Clean any old gem files
|
|
237
|
+
1. Clean any old gem files. **Note**: zsh fails on `rm -f *.gem` when no files match due to `nomatch`. Use:
|
|
238
|
+
```
|
|
239
|
+
find . -maxdepth 1 -name 'source_monitor-*.gem' -delete
|
|
240
|
+
```
|
|
218
241
|
2. Build the gem: `gem build source_monitor.gemspec`
|
|
219
242
|
3. Verify the gem was built: check for `source_monitor-X.Y.Z.gem` in the project root.
|
|
220
243
|
4. Show the file size: `ls -la source_monitor-X.Y.Z.gem`
|
|
221
244
|
|
|
222
|
-
## Step
|
|
245
|
+
## Step 12: Gem Push Instructions
|
|
223
246
|
|
|
224
247
|
Present the final instructions to the user:
|
|
225
248
|
|
data/.gitignore
CHANGED
|
@@ -22,3 +22,10 @@
|
|
|
22
22
|
.vbw-planning/.claude-md-migrated
|
|
23
23
|
.vbw-planning/.watchdog-pid
|
|
24
24
|
.vbw-planning/.watchdog.log
|
|
25
|
+
.vbw-planning/.agent-pids
|
|
26
|
+
.vbw-planning/.agent-panes
|
|
27
|
+
.vbw-planning/.active-agent
|
|
28
|
+
.vbw-planning/.active-agent-count
|
|
29
|
+
.vbw-planning/.todo-flat-migrated
|
|
30
|
+
/codebase_analysis.md
|
|
31
|
+
*.gem
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# Roadmap
|
|
2
|
+
|
|
3
|
+
## Milestone: aia-ssl-fix
|
|
4
|
+
|
|
5
|
+
### Phases
|
|
6
|
+
|
|
7
|
+
1. [x] **AIA Certificate Resolution** -- Fix SSL failures for feeds with missing intermediate certificates by implementing AIA (Authority Information Access) resolution
|
|
8
|
+
2. [x] **Test Performance** -- Reduce test suite runtime from ~133s to ~50s by splitting monolithic test classes, enabling parallelism, reducing log IO, and adopting before_all
|
|
9
|
+
|
|
10
|
+
### Phase Details
|
|
11
|
+
|
|
12
|
+
#### Phase 1: AIA Certificate Resolution
|
|
13
|
+
|
|
14
|
+
**Goal:** Implement automatic AIA intermediate certificate fetching so feeds like netflixtechblog.com (served via Medium/AWS with wrong intermediates) succeed without manual cert configuration.
|
|
15
|
+
|
|
16
|
+
**Requirements:**
|
|
17
|
+
- REQ-AIA-01: Create AIAResolver module with thread-safe cache and 1-hour TTL
|
|
18
|
+
- REQ-AIA-02: Add cert_store: parameter to HTTP.client for custom cert stores
|
|
19
|
+
- REQ-AIA-03: On Faraday::SSLError, attempt AIA resolution before failing
|
|
20
|
+
- REQ-AIA-04: Best-effort only -- never make things worse (rescue StandardError -> nil)
|
|
21
|
+
|
|
22
|
+
**Success Criteria:**
|
|
23
|
+
- [ ] AIAResolver.resolve(hostname) fetches leaf cert, extracts AIA URL, downloads intermediate
|
|
24
|
+
- [ ] HTTP.client(cert_store:) accepts and uses custom cert stores
|
|
25
|
+
- [ ] FeedFetcher retries once with AIA-resolved cert store on SSL failure
|
|
26
|
+
- [ ] All existing tests pass (1003+), new tests cover AIA paths
|
|
27
|
+
- [ ] RuboCop zero offenses, Brakeman zero warnings
|
|
28
|
+
|
|
29
|
+
#### Phase 2: Test Performance
|
|
30
|
+
|
|
31
|
+
**Goal:** Reduce test suite wall-clock time from ~133s to ~50s through structural optimizations. The 3-agent investigation identified that FeedFetcherTest (71 tests, 84.8s, 64% of total) is a monolithic class that cannot be parallelized, integration tests add 31s, and 95MB of debug logging adds 5-15s.
|
|
32
|
+
|
|
33
|
+
**Requirements:**
|
|
34
|
+
- REQ-PERF-01: Split FeedFetcherTest into 5+ smaller classes by concern (success paths, error handling, adaptive interval, dirty-check, content fingerprint, utilities)
|
|
35
|
+
- REQ-PERF-02: Set test log level to :warn in test/dummy/config/environments/test.rb (eliminates 95MB log IO)
|
|
36
|
+
- REQ-PERF-03: Tag integration tests (host_install_flow, release_packaging) so they can be excluded during dev with --exclude-pattern
|
|
37
|
+
- REQ-PERF-04: Switch default parallelism from forks to threads (avoids PG segfault, enables splitting benefit)
|
|
38
|
+
- REQ-PERF-05: Adopt before_all/setup_once in top DB-heavy test files (dashboard/queries_test.rb, etc.)
|
|
39
|
+
|
|
40
|
+
**Success Criteria:**
|
|
41
|
+
- [ ] FeedFetcherTest split into 5+ files, each independently runnable
|
|
42
|
+
- [ ] All 1031+ tests pass with PARALLEL_WORKERS=1 and default workers
|
|
43
|
+
- [ ] Test suite completes in <70s locally (down from 133s)
|
|
44
|
+
- [ ] `bin/rails test --exclude-pattern="**/integration/**"` runs <50s
|
|
45
|
+
- [ ] RuboCop zero offenses, Brakeman zero warnings
|
|
46
|
+
- [ ] No test isolation regressions (parallel runs still green)
|
|
47
|
+
|
|
48
|
+
### Progress
|
|
49
|
+
|
|
50
|
+
| Phase | Status | Plans | Completed |
|
|
51
|
+
|-------|--------|-------|-----------|
|
|
52
|
+
| 1. AIA Certificate Resolution | Complete | 3 | 3 |
|
|
53
|
+
| 2. Test Performance | Complete | 4 | 4 |
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# State
|
|
2
|
+
|
|
3
|
+
## Current Position
|
|
4
|
+
|
|
5
|
+
- **Milestone:** aia-ssl-fix
|
|
6
|
+
- **Phase:** 2 -- Test Performance
|
|
7
|
+
- **Status:** Complete
|
|
8
|
+
- **Progress:** 100%
|
|
9
|
+
|
|
10
|
+
## Decisions
|
|
11
|
+
|
|
12
|
+
| Decision | Date | Context |
|
|
13
|
+
|----------|------|---------|
|
|
14
|
+
| Single-phase milestone for AIA fix | 2026-02-17 | Complete plan already validated; no scoping needed |
|
|
15
|
+
| 3 plans with wave parallelism | 2026-02-17 | Plans 01+02 (wave 1, disjoint files), Plan 03 (wave 2, integration) |
|
|
16
|
+
|
|
17
|
+
## Todos
|
|
18
|
+
|
|
19
|
+
## Metrics
|
|
20
|
+
|
|
21
|
+
- **Started:** 2026-02-17
|
|
22
|
+
- **Phases:** 1
|
|
23
|
+
- **Plans:** 3
|
|
24
|
+
- **Tests at start:** 1003
|
|
25
|
+
- **Tests at end:** 1025
|
|
26
|
+
- **Commits:** 4 (f60e9bf, 4c9568a, 9c38bc3, e68a6b0)
|
|
27
|
+
- **Plans completed:** 3/3
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
## Phase 1 Context
|
|
2
|
+
|
|
3
|
+
### Goal
|
|
4
|
+
Not available
|
|
5
|
+
|
|
6
|
+
### Codebase Map Available
|
|
7
|
+
Codebase mapping exists in `.vbw-planning/codebase/`. Key files:
|
|
8
|
+
- `ARCHITECTURE.md`
|
|
9
|
+
- `CONCERNS.md`
|
|
10
|
+
- `PATTERNS.md`
|
|
11
|
+
- `DEPENDENCIES.md`
|
|
12
|
+
- `STRUCTURE.md`
|
|
13
|
+
- `CONVENTIONS.md`
|
|
14
|
+
- `TESTING.md`
|
|
15
|
+
- `STACK.md`
|
|
16
|
+
|
|
17
|
+
Read CONVENTIONS.md, PATTERNS.md, STRUCTURE.md, and DEPENDENCIES.md first to bootstrap codebase understanding.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
---
|
|
2
|
+
phase: 1
|
|
3
|
+
plan: 1
|
|
4
|
+
status: complete
|
|
5
|
+
---
|
|
6
|
+
# Plan 01 Summary: AIA Resolver Module
|
|
7
|
+
|
|
8
|
+
## Tasks Completed
|
|
9
|
+
- [x] Task 1: Created lib/source_monitor/http/aia_resolver.rb
|
|
10
|
+
- [x] Task 2: Created test/lib/source_monitor/http/aia_resolver_test.rb
|
|
11
|
+
|
|
12
|
+
## Commits
|
|
13
|
+
- 4c9568a: feat(1-1): add AIA intermediate certificate resolver
|
|
14
|
+
|
|
15
|
+
## Files Modified
|
|
16
|
+
- lib/source_monitor/http/aia_resolver.rb (created)
|
|
17
|
+
- test/lib/source_monitor/http/aia_resolver_test.rb (created)
|
|
18
|
+
|
|
19
|
+
## What Was Built
|
|
20
|
+
- `SourceMonitor::HTTP::AIAResolver` module with thread-safe cached resolution of missing intermediate SSL certificates via AIA (Authority Information Access) X.509 extension
|
|
21
|
+
- Public API: `resolve(hostname)`, `enhanced_cert_store(certs)`, `clear_cache!`, `cache_size`
|
|
22
|
+
- Private methods: `fetch_leaf_certificate` (VERIFY_NONE + SNI), `extract_aia_url` (uses `cert.ca_issuer_uris`), `download_certificate` (DER-first, PEM fallback)
|
|
23
|
+
- 11 unit tests covering all public/private methods, caching, TTL expiration, and error handling
|
|
24
|
+
|
|
25
|
+
## Deviations
|
|
26
|
+
- None
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
---
|
|
2
|
+
phase: 1
|
|
3
|
+
plan: 1
|
|
4
|
+
title: "AIA Resolver Module"
|
|
5
|
+
wave: 1
|
|
6
|
+
depends_on: []
|
|
7
|
+
must_haves:
|
|
8
|
+
- AIAResolver module with resolve, enhanced_cert_store, clear_cache!, cache_size
|
|
9
|
+
- Thread-safe Mutex + Hash cache with 1-hour TTL per hostname
|
|
10
|
+
- fetch_leaf_certificate with VERIFY_NONE and SNI support
|
|
11
|
+
- extract_aia_url using cert.ca_issuer_uris (not regex)
|
|
12
|
+
- download_certificate with DER-first, PEM-fallback parsing
|
|
13
|
+
- All methods rescue StandardError and return nil
|
|
14
|
+
- Unit tests covering all public and private methods
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
# Plan 01: AIA Resolver Module
|
|
18
|
+
|
|
19
|
+
## Goal
|
|
20
|
+
|
|
21
|
+
Create `SourceMonitor::HTTP::AIAResolver` -- a standalone module that resolves missing intermediate certificates via the AIA (Authority Information Access) extension in X.509 certificates.
|
|
22
|
+
|
|
23
|
+
## Tasks
|
|
24
|
+
|
|
25
|
+
### Task 1: Create lib/source_monitor/http/aia_resolver.rb
|
|
26
|
+
|
|
27
|
+
Create new module `SourceMonitor::HTTP::AIAResolver` with class methods:
|
|
28
|
+
|
|
29
|
+
**Public API:**
|
|
30
|
+
- `resolve(hostname, port: 443)` -- Entry point. Checks cache first, then: fetch leaf cert -> extract AIA URL -> download intermediate. Returns `OpenSSL::X509::Certificate` or `nil`.
|
|
31
|
+
- `enhanced_cert_store(additional_certs)` -- Builds `OpenSSL::X509::Store` with `set_default_paths` plus extra certs from the array.
|
|
32
|
+
- `clear_cache!` -- Clears the hostname cache (for testing).
|
|
33
|
+
- `cache_size` -- Returns number of cached entries (for testing).
|
|
34
|
+
|
|
35
|
+
**Private methods:**
|
|
36
|
+
- `fetch_leaf_certificate(hostname, port)` -- TCP+SSL connect with `VERIFY_NONE` to get the server's leaf cert. 5s connect timeout. Uses `ssl_socket.hostname=` for SNI.
|
|
37
|
+
- `extract_aia_url(cert)` -- Uses Ruby's built-in `cert.ca_issuer_uris` method. Returns first URI string or nil.
|
|
38
|
+
- `download_certificate(url)` -- Plain HTTP GET (AIA URLs are always HTTP, not HTTPS). 5s timeout. Parses DER body as `OpenSSL::X509::Certificate`, falls back to PEM on failure.
|
|
39
|
+
|
|
40
|
+
**Cache:** `Mutex` + `Hash` keyed by hostname. Each entry stores `{ cert:, expires_at: }` with 1-hour TTL.
|
|
41
|
+
|
|
42
|
+
**Safety:** All methods rescue `StandardError` and return `nil`. This is best-effort -- never makes things worse.
|
|
43
|
+
|
|
44
|
+
### Task 2: Create test/lib/source_monitor/http/aia_resolver_test.rb
|
|
45
|
+
|
|
46
|
+
Unit tests:
|
|
47
|
+
- `extract_aia_url` with cert that has AIA extension returns URL
|
|
48
|
+
- `extract_aia_url` with cert without AIA returns nil
|
|
49
|
+
- `download_certificate` with DER body parses correctly (WebMock stub)
|
|
50
|
+
- `download_certificate` returns nil on HTTP 404 (WebMock)
|
|
51
|
+
- `download_certificate` returns nil on timeout (WebMock)
|
|
52
|
+
- `enhanced_cert_store` returns store with added certs
|
|
53
|
+
- `enhanced_cert_store` handles empty array gracefully
|
|
54
|
+
- Cache: resolve stores result, second call returns cached
|
|
55
|
+
- Cache: expired entries are re-fetched
|
|
56
|
+
- `clear_cache!` empties the cache
|
|
57
|
+
- `resolve` returns nil when hostname unreachable (stub fetch_leaf_certificate)
|
|
58
|
+
|
|
59
|
+
## Files
|
|
60
|
+
|
|
61
|
+
| Action | Path |
|
|
62
|
+
|--------|------|
|
|
63
|
+
| CREATE | `lib/source_monitor/http/aia_resolver.rb` |
|
|
64
|
+
| CREATE | `test/lib/source_monitor/http/aia_resolver_test.rb` |
|
|
65
|
+
|
|
66
|
+
## Verification
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/http/aia_resolver_test.rb
|
|
70
|
+
bin/rubocop lib/source_monitor/http/aia_resolver.rb test/lib/source_monitor/http/aia_resolver_test.rb
|
|
71
|
+
```
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
---
|
|
2
|
+
phase: 1
|
|
3
|
+
plan: 2
|
|
4
|
+
status: complete
|
|
5
|
+
commit: f60e9bf
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## What Was Built
|
|
9
|
+
- Added `cert_store:` keyword parameter to `HTTP.client` for custom OpenSSL cert stores
|
|
10
|
+
- Added `autoload :AIAResolver` to HTTP module
|
|
11
|
+
- Plumbed cert_store through `configure_request` -> `configure_ssl` with fallback to `default_cert_store`
|
|
12
|
+
- 2 new tests: custom cert_store usage, ssl_ca_file takes precedence over cert_store
|
|
13
|
+
|
|
14
|
+
## Files Modified
|
|
15
|
+
- `lib/source_monitor/http.rb` — autoload, cert_store param, SSL plumbing
|
|
16
|
+
- `test/lib/source_monitor/http_test.rb` — 2 new cert_store tests
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
---
|
|
2
|
+
phase: 1
|
|
3
|
+
plan: 2
|
|
4
|
+
title: "HTTP Module cert_store Parameter"
|
|
5
|
+
wave: 1
|
|
6
|
+
depends_on: []
|
|
7
|
+
must_haves:
|
|
8
|
+
- Add autoload :AIAResolver to module HTTP
|
|
9
|
+
- Add cert_store keyword to client method
|
|
10
|
+
- Pass cert_store through configure_request to configure_ssl
|
|
11
|
+
- configure_ssl uses cert_store when no ssl_ca_file/ssl_ca_path
|
|
12
|
+
- Tests for cert_store parameter usage
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
# Plan 02: HTTP Module cert_store Parameter
|
|
16
|
+
|
|
17
|
+
## Goal
|
|
18
|
+
|
|
19
|
+
Extend `SourceMonitor::HTTP.client` to accept an optional `cert_store:` parameter, enabling callers (like FeedFetcher's AIA retry) to provide a custom `OpenSSL::X509::Store` with additional certificates.
|
|
20
|
+
|
|
21
|
+
## Tasks
|
|
22
|
+
|
|
23
|
+
### Task 1: Modify lib/source_monitor/http.rb
|
|
24
|
+
|
|
25
|
+
1. Add autoload inside `module HTTP` (after RETRY_STATUSES):
|
|
26
|
+
```ruby
|
|
27
|
+
autoload :AIAResolver, "source_monitor/http/aia_resolver"
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
2. Add `cert_store: nil` keyword to `client` method signature.
|
|
31
|
+
|
|
32
|
+
3. Pass `cert_store:` through `configure_request` to `configure_ssl`:
|
|
33
|
+
- Add `cert_store:` parameter to `configure_request`
|
|
34
|
+
- Pass it to `configure_ssl(connection, settings, cert_store:)`
|
|
35
|
+
|
|
36
|
+
4. In `configure_ssl`: when no `ssl_ca_file` or `ssl_ca_path` is set, use `cert_store || default_cert_store`.
|
|
37
|
+
|
|
38
|
+
### Task 2: Add tests to test/lib/source_monitor/http_test.rb
|
|
39
|
+
|
|
40
|
+
Add 2 tests:
|
|
41
|
+
- `cert_store: param is used when no ssl_ca_file or ssl_ca_path` -- pass a custom store, verify `connection.ssl.cert_store` is the custom store
|
|
42
|
+
- `cert_store: is ignored when ssl_ca_file is set` -- configure ssl_ca_file, pass cert_store, verify ca_file takes precedence
|
|
43
|
+
|
|
44
|
+
## Files
|
|
45
|
+
|
|
46
|
+
| Action | Path |
|
|
47
|
+
|--------|------|
|
|
48
|
+
| MODIFY | `lib/source_monitor/http.rb` |
|
|
49
|
+
| MODIFY | `test/lib/source_monitor/http_test.rb` |
|
|
50
|
+
|
|
51
|
+
## Verification
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/http_test.rb
|
|
55
|
+
bin/rubocop lib/source_monitor/http.rb test/lib/source_monitor/http_test.rb
|
|
56
|
+
```
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
---
|
|
2
|
+
phase: 1
|
|
3
|
+
plan: 3
|
|
4
|
+
status: complete
|
|
5
|
+
commit: 9c38bc3
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## What Was Built
|
|
9
|
+
- Wired AIA certificate resolution into FeedFetcher's SSL error handling
|
|
10
|
+
- On `Faraday::SSLError`, attempts intermediate cert recovery via `AIAResolver.resolve` before raising
|
|
11
|
+
- Guard flag `@aia_attempted` prevents infinite recursion; `rescue StandardError => nil` ensures recovery never makes things worse
|
|
12
|
+
- Tags `instrumentation_payload[:aia_resolved] = true` on successful AIA recovery
|
|
13
|
+
- 3 integration tests: success retry path, nil fallback to ConnectionError, non-SSL skip
|
|
14
|
+
|
|
15
|
+
## Files Modified
|
|
16
|
+
- `lib/source_monitor/fetching/feed_fetcher.rb` — split SSL rescue, add `attempt_aia_recovery`
|
|
17
|
+
- `test/lib/source_monitor/fetching/feed_fetcher_test.rb` — 3 AIA resolution tests
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
---
|
|
2
|
+
phase: 1
|
|
3
|
+
plan: 3
|
|
4
|
+
title: "FeedFetcher AIA Retry Integration"
|
|
5
|
+
wave: 2
|
|
6
|
+
depends_on: [1, 2]
|
|
7
|
+
must_haves:
|
|
8
|
+
- Separate Faraday::SSLError rescue from Faraday::ConnectionFailed
|
|
9
|
+
- On SSLError attempt AIA resolution once (aia_attempted flag)
|
|
10
|
+
- Parse hostname from source.feed_url for AIA resolve
|
|
11
|
+
- If intermediate found rebuild connection with enhanced cert store and retry
|
|
12
|
+
- If nil raise ConnectionError as before
|
|
13
|
+
- Tag successful recoveries with aia_resolved in instrumentation
|
|
14
|
+
- Integration tests for all AIA retry paths
|
|
15
|
+
- Full test suite passes (1003+ tests)
|
|
16
|
+
- RuboCop zero offenses
|
|
17
|
+
- Brakeman zero warnings
|
|
18
|
+
---
|
|
19
|
+
|
|
20
|
+
# Plan 03: FeedFetcher AIA Retry Integration
|
|
21
|
+
|
|
22
|
+
## Goal
|
|
23
|
+
|
|
24
|
+
Wire AIA resolution into FeedFetcher's error handling so SSL failures automatically attempt intermediate certificate recovery before giving up.
|
|
25
|
+
|
|
26
|
+
## Tasks
|
|
27
|
+
|
|
28
|
+
### Task 1: Modify lib/source_monitor/fetching/feed_fetcher.rb
|
|
29
|
+
|
|
30
|
+
Modify `perform_fetch` (lines 77-90):
|
|
31
|
+
|
|
32
|
+
1. **Split rescue clause:** Separate `Faraday::SSLError` from `Faraday::ConnectionFailed` into its own rescue:
|
|
33
|
+
```ruby
|
|
34
|
+
rescue Faraday::ConnectionFailed => error
|
|
35
|
+
raise ConnectionError.new(error.message, original_error: error)
|
|
36
|
+
rescue Faraday::SSLError => error
|
|
37
|
+
attempt_aia_recovery(error) || raise(ConnectionError.new(error.message, original_error: error))
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
2. **Add `attempt_aia_recovery` private method:**
|
|
41
|
+
- Guard: return nil if `@aia_attempted` is true (prevents recursion)
|
|
42
|
+
- Set `@aia_attempted = true`
|
|
43
|
+
- Parse hostname from `URI.parse(source.feed_url).host`
|
|
44
|
+
- Call `SourceMonitor::HTTP::AIAResolver.resolve(hostname)`
|
|
45
|
+
- If intermediate found:
|
|
46
|
+
- Build enhanced cert store via `AIAResolver.enhanced_cert_store([intermediate])`
|
|
47
|
+
- Rebuild `@connection = SourceMonitor::HTTP.client(cert_store: store, headers: request_headers)`
|
|
48
|
+
- Return `perform_request` (the retry)
|
|
49
|
+
- If nil: return nil (caller raises ConnectionError)
|
|
50
|
+
- Rescue StandardError -> nil (never make retry worse)
|
|
51
|
+
|
|
52
|
+
3. **Tag instrumentation:** In the `handle_response` path after successful AIA retry, the `instrumentation_payload[:aia_resolved] = true` will naturally flow through since `perform_fetch` calls `handle_response` on the retried response.
|
|
53
|
+
|
|
54
|
+
### Task 2: Add tests to test/lib/source_monitor/fetching/feed_fetcher_test.rb
|
|
55
|
+
|
|
56
|
+
Add 3 tests under a new section `# -- AIA Certificate Resolution --`:
|
|
57
|
+
|
|
58
|
+
1. **SSL error + AIA resolve succeeds -> fetch succeeds:**
|
|
59
|
+
- First stub: raise `Faraday::SSLError`
|
|
60
|
+
- Stub `AIAResolver.resolve` to return a mock certificate
|
|
61
|
+
- Stub `AIAResolver.enhanced_cert_store` to return a store
|
|
62
|
+
- Second stub (after retry): return 200 with RSS body
|
|
63
|
+
- Assert result.status == :fetched
|
|
64
|
+
|
|
65
|
+
2. **SSL error + AIA resolve returns nil -> ConnectionError:**
|
|
66
|
+
- Stub to raise `Faraday::SSLError`
|
|
67
|
+
- Stub `AIAResolver.resolve` to return nil
|
|
68
|
+
- Assert result.status == :failed
|
|
69
|
+
- Assert result.error is ConnectionError
|
|
70
|
+
|
|
71
|
+
3. **Non-SSL ConnectionError -> AIA not attempted:**
|
|
72
|
+
- Stub to raise `Faraday::ConnectionFailed`
|
|
73
|
+
- Verify `AIAResolver.resolve` was NOT called
|
|
74
|
+
- Assert result.status == :failed
|
|
75
|
+
- Assert result.error is ConnectionError
|
|
76
|
+
|
|
77
|
+
### Task 3: Run full verification
|
|
78
|
+
|
|
79
|
+
1. `PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb`
|
|
80
|
+
2. `bin/rails test` (full suite)
|
|
81
|
+
3. `bin/rubocop`
|
|
82
|
+
4. `bin/brakeman --no-pager`
|
|
83
|
+
|
|
84
|
+
## Files
|
|
85
|
+
|
|
86
|
+
| Action | Path |
|
|
87
|
+
|--------|------|
|
|
88
|
+
| MODIFY | `lib/source_monitor/fetching/feed_fetcher.rb` |
|
|
89
|
+
| MODIFY | `test/lib/source_monitor/fetching/feed_fetcher_test.rb` |
|
|
90
|
+
|
|
91
|
+
## Verification
|
|
92
|
+
|
|
93
|
+
```bash
|
|
94
|
+
PARALLEL_WORKERS=1 bin/rails test test/lib/source_monitor/fetching/feed_fetcher_test.rb
|
|
95
|
+
bin/rails test
|
|
96
|
+
bin/rubocop
|
|
97
|
+
bin/brakeman --no-pager
|
|
98
|
+
```
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
## Phase 02 Context
|
|
2
|
+
|
|
3
|
+
### Goal
|
|
4
|
+
Not available
|
|
5
|
+
|
|
6
|
+
### Codebase Map Available
|
|
7
|
+
Codebase mapping exists in `.vbw-planning/codebase/`. Key files:
|
|
8
|
+
- `ARCHITECTURE.md`
|
|
9
|
+
- `CONCERNS.md`
|
|
10
|
+
- `PATTERNS.md`
|
|
11
|
+
- `DEPENDENCIES.md`
|
|
12
|
+
- `STRUCTURE.md`
|
|
13
|
+
- `CONVENTIONS.md`
|
|
14
|
+
- `TESTING.md`
|
|
15
|
+
- `STACK.md`
|
|
16
|
+
|
|
17
|
+
Read CONVENTIONS.md, PATTERNS.md, STRUCTURE.md, and DEPENDENCIES.md first to bootstrap codebase understanding.
|
|
18
|
+
|
|
19
|
+
### Research Findings
|
|
20
|
+
# Phase 2: Test Performance - Research
|
|
21
|
+
|
|
22
|
+
## Investigation Method
|
|
23
|
+
3-agent competing hypothesis team (H1: DB/parallelism, H2: HTTP/VCR, H3: test helpers)
|
|
24
|
+
|
|
25
|
+
## Findings
|
|
26
|
+
|
|
27
|
+
### Time Budget (133s total, 1031 tests)
|
|
28
|
+
| Component | Time | % | Root Cause |
|
|
29
|
+
|-----------|------|---|------------|
|
|
30
|
+
| FeedFetcherTest (71 tests) | 84.8s | 64% | Monolithic class, CPU-bound (Feedjira parse, SHA256, content extraction) |
|
|
31
|
+
| Integration tests (9 tests) | 30.8s | 23% | Subprocess spawning (bundle exec rails g) |
|
|
32
|
+
| System tests (22 tests) | ~5s | 4% | Selenium/Chrome |
|
|
33
|
+
| Debug log IO | 5-15s | 8% | 95MB :debug output to disk |
|
|
34
|
+
| Everything else (~930 tests) | ~15s | 11% | Fast |
|
|
35
|
+
|
|
36
|
+
### Why Parallelism Doesn't Help Today
|
|
37
|
+
- Minitest distributes by CLASS, not by test
|
|
38
|
+
- FeedFetcherTest = 1 class with 71 tests = 1 worker gets all 84.8s
|
|
39
|
+
- With 8 workers: max(84.8, 25.6, ~3) ≈ 85s + overhead
|
|
40
|
+
|
|
41
|
+
### DB Is NOT the Bottleneck
|
|
42
|
+
- SQL: 3.639s of 69.4s (5.25%) via EventProf
|
|
43
|
+
- schema.rb used (fast load)
|
|
44
|
+
- Transactional tests enabled (proper rollback)
|
|
45
|
+
|
|
46
|
+
### HTTP Mocking Is NOT the Bottleneck
|
|
47
|
+
- Only 4 VCR cassettes (456KB total), 83 WebMock stubs across 9 files
|
|
48
|
+
- WebMock configured once globally, auto-resets
|
|
49
|
+
- Only finding: default_cert_store recreated per connection (~0.5s total)
|
|
50
|
+
|
|
51
|
+
### Test Helpers Are Cheap
|
|
52
|
+
- reset_configuration!: microseconds per call (pure Ruby)
|
|
53
|
+
- create_source!: 1 INSERT per call, 158 calls total
|
|
54
|
+
- with_inline_jobs: 4 calls total, negligible
|
|
55
|
+
|
|
56
|
+
### Profiling Data (tmp/test_prof/)
|
|
57
|
+
- TagProf: lib tests 38.0s (55%), integration 28.5s (41%), controllers 0.97s, jobs 0.53s, models 0.47s
|
|
58
|
+
- EventProf: FeedFetcherTest 0.888s SQL in 34.3s total (2.6% SQL)
|
|
59
|
+
|
|
60
|
+
## Relevant Patterns
|
|
61
|
+
- TestProf before_all available but used in only 1 of 54 eligible files
|
|
62
|
+
- Thread-based parallelism already proven in coverage mode
|
|
63
|
+
- PG fork segfault only on single-file runs, NOT full suite
|
|
64
|
+
|
|
65
|
+
## Risks
|
|
66
|
+
- Thread safety: reset_configuration! may need thread-local config or mutex
|
|
67
|
+
- Test isolation: splitting FeedFetcherTest must preserve test independence
|
|
68
|
+
- before_all: only safe for tests that don't mutate shared data
|
|
69
|
+
|
|
70
|
+
## Recommendations (by impact)
|
|
71
|
+
1. Split FeedFetcherTest into 5-6 classes → enables parallelism, -50s
|
|
72
|
+
2. Set config.log_level = :warn → -5-15s
|
|
73
|
+
3. Tag/skip integration tests in dev → -30s dev experience
|
|
74
|
+
4. Switch to thread-based parallelism → enables splitting benefit
|
|
75
|
+
5. Adopt before_all in DB-heavy test files → -3-5s
|