traject-solr_pool 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.github/workflows/ci.yml +34 -0
- data/CLAUDE.md +237 -0
- data/LICENSE.txt +21 -0
- data/README.md +167 -0
- data/Rakefile +21 -0
- data/docs/agents/README.md +41 -0
- data/docs/skills/README.md +42 -0
- data/docs/superpowers/plans/2026-07-01-traject-solr-pool-writer.md +1387 -0
- data/docs/superpowers/specs/2026-07-01-traject-solr-pool-writer-design.md +265 -0
- data/docs/usage.md +122 -0
- data/lib/traject/solr_pool/connection.rb +80 -0
- data/lib/traject/solr_pool/solr_json_writer.rb +303 -0
- data/lib/traject/solr_pool/version.rb +7 -0
- data/lib/traject/solr_pool.rb +15 -0
- data/sig/traject/solr_pool.rbs +6 -0
- metadata +95 -0
|
@@ -0,0 +1,1387 @@
|
|
|
1
|
+
# Traject Solr Pool Writer Implementation Plan
|
|
2
|
+
|
|
3
|
+
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
4
|
+
|
|
5
|
+
**Goal:** Build `Traject::SolrPool::SolrJsonWriter`, a drop-in replacement for `Traject::SolrJsonWriter` that routes every Solr HTTP call through a persistent, credential-isolated connection pool from `http_connection_pool`.
|
|
6
|
+
|
|
7
|
+
**Architecture:** Two units. `Traject::SolrPool::Connection` owns all pooling (derives the origin, builds `pool_options`, binds to the pool by extending `HttpConnectionPool::Connectable` onto an adapter object) and exposes `post`/`get`. `Traject::SolrPool::SolrJsonWriter` is a fresh port of the stock writer's structure (batched queue, `Traject::ThreadPool`, skip counter, `commit`/`delete`) that delegates all HTTP to `Connection`. The `Connection` seam is designed for a future reader to reuse.
|
|
8
|
+
|
|
9
|
+
**Tech Stack:** Ruby 3.4 (floor 3.3), RSpec, WebMock, http.rb 6.x, `http_connection_pool`, traject (edge via Gemfile `path:`), concurrent-ruby.
|
|
10
|
+
|
|
11
|
+
## Global Constraints
|
|
12
|
+
|
|
13
|
+
- Ruby `>= 3.3.0`; target and test on MRI (CRuby). Do not claim JRuby support.
|
|
14
|
+
- Runtime deps only: `http_connection_pool ~> 0.1`, `traject` (edge via Gemfile `path:` override until a release lifts the `http < 6` cap; gemspec constraint carries a NOTE to switch to the released version). Never add Rails/Zeitwerk/Sidekiq/WebMock to `spec.add_dependency`.
|
|
15
|
+
- Test-only deps live in a `:test` Gemfile group (NOT `:development`): `webmock`, `activesupport`, `activejob`, `zeitwerk`, `sidekiq`.
|
|
16
|
+
- All non-interpolated Ruby strings use single quotes. Every `.rb` file starts with `# frozen_string_literal: true`.
|
|
17
|
+
- RuboCop (`single_quotes`, plugins `rubocop-performance`/`rubocop-rake`/`rubocop-rspec`, `TargetRubyVersion 3.3`) must be clean before every commit. No inline `# rubocop:disable` unless unavoidable.
|
|
18
|
+
- Reuse the `solr.*` / `solr_writer.*` settings vocabulary verbatim. Only new setting is `solr_pool.pool_size`.
|
|
19
|
+
- Never log or raise a message containing basic-auth credentials or auth headers. Origin only in logs.
|
|
20
|
+
- Pooling is delegated to `http_connection_pool` via its public API (`Connectable` / `Registry`). Never reach into its internals or reimplement keep-alive/sockets/mutexes.
|
|
21
|
+
- Thread safety is mandatory: `concurrent-ruby` primitives, no coarse global `Mutex` on the hot path. A borrowed connection is never held across threads.
|
|
22
|
+
- RSpec: no apostrophes in `it`/`describe`/`context` strings; `after do` teardown; reset `HttpConnectionPool::Registry` between examples; `spec/support/**` auto-required; helpers in a `spec/support` module included by tag.
|
|
23
|
+
- Git: commit freely, NEVER push (also no `gem push`/`rake release`). Stage files by name, never `git add -A`/`.`; never stage `Gemfile.lock` (gitignored).
|
|
24
|
+
|
|
25
|
+
**Reference:** design spec at `docs/superpowers/specs/2026-07-01-traject-solr-pool-writer-design.md`.
|
|
26
|
+
|
|
27
|
+
---
|
|
28
|
+
|
|
29
|
+
## File Structure
|
|
30
|
+
|
|
31
|
+
- `lib/traject/solr_pool.rb` — entry point; requires version, connection, writer.
|
|
32
|
+
- `lib/traject/solr_pool/version.rb` — `Traject::SolrPool::VERSION` (exists).
|
|
33
|
+
- `lib/traject/solr_pool/connection.rb` — `Traject::SolrPool::Connection` (reusable pool seam).
|
|
34
|
+
- `lib/traject/solr_pool/solr_json_writer.rb` — `Traject::SolrPool::SolrJsonWriter` + nested `BadHttpResponse`, `MaxSkippedRecordsExceeded`.
|
|
35
|
+
- `spec/spec_helper.rb` — require gem, WebMock config, Registry reset, support autoload.
|
|
36
|
+
- `spec/support/webmock_helpers.rb` — Solr stub helpers, tag `:solr_stub`.
|
|
37
|
+
- `spec/support/thread_safety_helpers.rb` — `cyclic_barrier`, tag `:thread_safety`.
|
|
38
|
+
- `spec/support/job_helpers.rb` — Sidekiq/ActiveJob harness, tag `:background_jobs`.
|
|
39
|
+
- `spec/traject/solr_pool/connection_spec.rb`
|
|
40
|
+
- `spec/traject/solr_pool/solr_json_writer_spec.rb`
|
|
41
|
+
- `spec/integration/concurrency_spec.rb`
|
|
42
|
+
- `spec/integration/background_job_spec.rb`
|
|
43
|
+
- `spec/integration/zeitwerk_compliance_spec.rb`
|
|
44
|
+
- `spec/integration/rails_compatibility_spec.rb`
|
|
45
|
+
|
|
46
|
+
---
|
|
47
|
+
|
|
48
|
+
## Task 1: Project scaffolding — deps, RuboCop, spec_helper, WebMock
|
|
49
|
+
|
|
50
|
+
**Files:**
|
|
51
|
+
- Modify: `traject-solr_pool.gemspec`
|
|
52
|
+
- Modify: `Gemfile`
|
|
53
|
+
- Modify: `spec/spec_helper.rb`
|
|
54
|
+
- Create: `spec/support/webmock_helpers.rb`
|
|
55
|
+
- Delete existing placeholder assertion in: `spec/traject/solr_pool_spec.rb`
|
|
56
|
+
|
|
57
|
+
**Interfaces:**
|
|
58
|
+
- Produces: a green `bundle exec rspec` + `bundle exec rubocop` baseline; `WebmockHelpers#stub_solr_update(status:, body:)` and `#stub_solr_get(...)` for later tasks; `spec_helper` resets `HttpConnectionPool::Registry.instance` after each example.
|
|
59
|
+
|
|
60
|
+
- [ ] **Step 1: Fill in the gemspec metadata and dependencies**
|
|
61
|
+
|
|
62
|
+
Replace the TODO/placeholder lines in `traject-solr_pool.gemspec`. Keep the existing `git ls-files` file list block. Set:
|
|
63
|
+
|
|
64
|
+
```ruby
|
|
65
|
+
spec.summary = 'Traject Solr JSON writer backed by a persistent, pooled, thread-safe HTTP client'
|
|
66
|
+
spec.description = 'A traject plugin providing a drop-in Solr JSON writer that routes updates ' \
|
|
67
|
+
'through http_connection_pool for persistent, credential-isolated, thread-safe connections.'
|
|
68
|
+
spec.homepage = 'https://github.com/bbarberBPL/traject-solr_pool'
|
|
69
|
+
spec.license = 'MIT'
|
|
70
|
+
|
|
71
|
+
spec.required_ruby_version = '>= 3.3.0'
|
|
72
|
+
|
|
73
|
+
spec.metadata['allowed_push_host'] = 'https://rubygems.org'
|
|
74
|
+
spec.metadata['homepage_uri'] = spec.homepage
|
|
75
|
+
spec.metadata['source_code_uri'] = spec.homepage
|
|
76
|
+
spec.metadata['rubygems_mfa_required'] = 'true'
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Keep the two runtime deps already present, but correct the traject line's NOTE:
|
|
80
|
+
|
|
81
|
+
```ruby
|
|
82
|
+
spec.add_dependency 'http_connection_pool', '~> 0.1'
|
|
83
|
+
# NOTE: temporary floor. The last RELEASED traject caps http < 6, colliding
|
|
84
|
+
# with http_connection_pool's http ~> 6.0. We depend on edge traject via a
|
|
85
|
+
# Gemfile `path:` override until a release lifts that cap; then pin the real
|
|
86
|
+
# released version here and remove the Gemfile override.
|
|
87
|
+
spec.add_dependency 'traject', '>= 3.8.4', '~> 3.8'
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
- [ ] **Step 2: Rewrite the Gemfile test group**
|
|
91
|
+
|
|
92
|
+
Replace the `group :development, :test` block in `Gemfile` with a split: keep tooling in `:development, :test` but put the integration-only libraries in a `:test`-only group.
|
|
93
|
+
|
|
94
|
+
```ruby
|
|
95
|
+
group :development, :test do
|
|
96
|
+
gem 'bundler-audit', '~> 0.9', require: false
|
|
97
|
+
gem 'irb', '~> 1.14'
|
|
98
|
+
gem 'rake', '~> 13.0'
|
|
99
|
+
gem 'rspec', '~> 3.13'
|
|
100
|
+
gem 'rubocop', require: false
|
|
101
|
+
gem 'rubocop-performance', require: false
|
|
102
|
+
gem 'rubocop-rake', require: false
|
|
103
|
+
gem 'rubocop-rspec', require: false
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
group :test do
|
|
107
|
+
gem 'webmock', '~> 3.23'
|
|
108
|
+
gem 'activesupport', '~> 7.2'
|
|
109
|
+
gem 'activejob', '~> 7.2'
|
|
110
|
+
gem 'zeitwerk', '~> 2.6'
|
|
111
|
+
gem 'sidekiq', '>= 8', '< 9'
|
|
112
|
+
end
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
Leave the existing `gem 'traject', path: '../traject-edge'` line as-is (the temporary edge override).
|
|
116
|
+
|
|
117
|
+
- [ ] **Step 3: Run bundle install**
|
|
118
|
+
|
|
119
|
+
Run: `bundle install`
|
|
120
|
+
Expected: resolves successfully, writes `Gemfile.lock` (gitignored — do not stage it).
|
|
121
|
+
|
|
122
|
+
- [ ] **Step 4: Write the WebMock helper module**
|
|
123
|
+
|
|
124
|
+
Create `spec/support/webmock_helpers.rb`:
|
|
125
|
+
|
|
126
|
+
```ruby
|
|
127
|
+
# frozen_string_literal: true
|
|
128
|
+
|
|
129
|
+
require 'webmock/rspec'
|
|
130
|
+
|
|
131
|
+
# HTTP stubbing helpers for Solr update/get endpoints. Included into any
|
|
132
|
+
# example group tagged :solr_stub (wired in spec_helper.rb). WebMock intercepts
|
|
133
|
+
# real http.rb requests, so specs exercise the actual pool without a live Solr.
|
|
134
|
+
module WebmockHelpers
|
|
135
|
+
SOLR_OK = { 'responseHeader' => { 'status' => 0, 'QTime' => 1 } }.freeze
|
|
136
|
+
|
|
137
|
+
def stub_solr_update(url, status: 200, body: SOLR_OK.to_json)
|
|
138
|
+
stub_request(:post, url).to_return(
|
|
139
|
+
status: status,
|
|
140
|
+
body: body,
|
|
141
|
+
headers: { 'Content-Type' => 'application/json' }
|
|
142
|
+
)
|
|
143
|
+
end
|
|
144
|
+
|
|
145
|
+
def stub_solr_get(url, status: 200, body: '{}')
|
|
146
|
+
stub_request(:get, url).to_return(
|
|
147
|
+
status: status,
|
|
148
|
+
body: body,
|
|
149
|
+
headers: { 'Content-Type' => 'application/json' }
|
|
150
|
+
)
|
|
151
|
+
end
|
|
152
|
+
end
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
- [ ] **Step 5: Rewrite spec_helper.rb**
|
|
156
|
+
|
|
157
|
+
Replace `spec/spec_helper.rb` with:
|
|
158
|
+
|
|
159
|
+
```ruby
|
|
160
|
+
# frozen_string_literal: true
|
|
161
|
+
|
|
162
|
+
require 'traject/solr_pool'
|
|
163
|
+
require 'webmock/rspec'
|
|
164
|
+
|
|
165
|
+
Dir[File.join(__dir__, 'support', '**', '*.rb')].each { |f| require f }
|
|
166
|
+
|
|
167
|
+
RSpec.configure do |config|
|
|
168
|
+
config.example_status_persistence_file_path = '.rspec_status'
|
|
169
|
+
config.disable_monkey_patching!
|
|
170
|
+
|
|
171
|
+
config.expect_with :rspec do |c|
|
|
172
|
+
c.syntax = :expect
|
|
173
|
+
end
|
|
174
|
+
|
|
175
|
+
config.include WebmockHelpers, :solr_stub
|
|
176
|
+
|
|
177
|
+
# Every example starts with a clean global pool registry so pools never leak
|
|
178
|
+
# between examples (locally-built pools must be closed by their own example).
|
|
179
|
+
config.after do
|
|
180
|
+
HttpConnectionPool::Registry.instance.close_all
|
|
181
|
+
end
|
|
182
|
+
end
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
- [ ] **Step 6: Replace the placeholder version spec**
|
|
186
|
+
|
|
187
|
+
Overwrite `spec/traject/solr_pool_spec.rb` (removing the failing `expect(false).to eq(true)` placeholder):
|
|
188
|
+
|
|
189
|
+
```ruby
|
|
190
|
+
# frozen_string_literal: true
|
|
191
|
+
|
|
192
|
+
RSpec.describe Traject::SolrPool do
|
|
193
|
+
it 'has a version number' do
|
|
194
|
+
expect(Traject::SolrPool::VERSION).not_to be_nil
|
|
195
|
+
end
|
|
196
|
+
end
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
- [ ] **Step 7: Run the suite and RuboCop**
|
|
200
|
+
|
|
201
|
+
Run: `bundle exec rspec && bundle exec rubocop`
|
|
202
|
+
Expected: RSpec green (1 example), RuboCop clean. Confirm `HttpConnectionPool::Registry` responds to `close_all` (it is the documented teardown method); if the version installed uses `reset!`, use `HttpConnectionPool::Registry.reset!` instead.
|
|
203
|
+
|
|
204
|
+
- [ ] **Step 8: Commit**
|
|
205
|
+
|
|
206
|
+
```bash
|
|
207
|
+
git add traject-solr_pool.gemspec Gemfile spec/spec_helper.rb spec/support/webmock_helpers.rb spec/traject/solr_pool_spec.rb
|
|
208
|
+
git commit -m "Scaffold deps, RuboCop, and WebMock-backed spec harness"
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
---
|
|
212
|
+
|
|
213
|
+
## Task 2: Connection — origin derivation and pool binding
|
|
214
|
+
|
|
215
|
+
**Files:**
|
|
216
|
+
- Create: `lib/traject/solr_pool/connection.rb`
|
|
217
|
+
- Create: `spec/traject/solr_pool/connection_spec.rb`
|
|
218
|
+
- Modify: `lib/traject/solr_pool.rb`
|
|
219
|
+
|
|
220
|
+
**Interfaces:**
|
|
221
|
+
- Consumes: `HttpConnectionPool::Connectable`, `HttpConnectionPool::Registry`.
|
|
222
|
+
- Produces: `Traject::SolrPool::Connection.new(origin:, pool_size:, pool_timeout:, headers: {}, auth: nil, timeout: nil)`; readers `#origin`; methods `#post(path, body:)` and `#get(path, params: {})` returning an http.rb response (`HTTP::Response`); `#release` closing this connection's pool via the registry.
|
|
223
|
+
|
|
224
|
+
- [ ] **Step 1: Write failing tests for origin binding and requests**
|
|
225
|
+
|
|
226
|
+
Create `spec/traject/solr_pool/connection_spec.rb`:
|
|
227
|
+
|
|
228
|
+
```ruby
|
|
229
|
+
# frozen_string_literal: true
|
|
230
|
+
|
|
231
|
+
RSpec.describe Traject::SolrPool::Connection, :solr_stub do
|
|
232
|
+
subject(:connection) { described_class.new(origin: 'http://solr.test:8983', pool_size: 2) }
|
|
233
|
+
|
|
234
|
+
it 'exposes the origin it was built with' do
|
|
235
|
+
expect(connection.origin).to eq('http://solr.test:8983')
|
|
236
|
+
end
|
|
237
|
+
|
|
238
|
+
it 'posts a body to a relative path resolved against the origin' do
|
|
239
|
+
stub = stub_solr_update('http://solr.test:8983/solr/core/update')
|
|
240
|
+
connection.post('/solr/core/update', body: '[]')
|
|
241
|
+
expect(stub).to have_been_requested
|
|
242
|
+
end
|
|
243
|
+
|
|
244
|
+
it 'gets a relative path with query params resolved against the origin' do
|
|
245
|
+
stub = stub_solr_get('http://solr.test:8983/solr/core/get?id=abc')
|
|
246
|
+
connection.get('/solr/core/get', params: { id: 'abc' })
|
|
247
|
+
expect(stub).to have_been_requested
|
|
248
|
+
end
|
|
249
|
+
|
|
250
|
+
it 'sends the configured auth as an Authorization header' do
|
|
251
|
+
connection = described_class.new(origin: 'http://solr.test:8983', pool_size: 1,
|
|
252
|
+
auth: 'Basic dXNlcjpwYXNz')
|
|
253
|
+
stub = stub_solr_update('http://solr.test:8983/x')
|
|
254
|
+
.with(headers: { 'Authorization' => 'Basic dXNlcjpwYXNz' })
|
|
255
|
+
connection.post('/x', body: '[]')
|
|
256
|
+
expect(stub).to have_been_requested
|
|
257
|
+
end
|
|
258
|
+
|
|
259
|
+
it 'gives different credentials different pools' do
|
|
260
|
+
a = described_class.new(origin: 'http://solr.test:8983', pool_size: 1, auth: 'Basic AAA')
|
|
261
|
+
b = described_class.new(origin: 'http://solr.test:8983', pool_size: 1, auth: 'Basic BBB')
|
|
262
|
+
expect(a.send(:pool)).not_to be(b.send(:pool))
|
|
263
|
+
end
|
|
264
|
+
end
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
- [ ] **Step 2: Run to verify failure**
|
|
268
|
+
|
|
269
|
+
Run: `bundle exec rspec spec/traject/solr_pool/connection_spec.rb`
|
|
270
|
+
Expected: FAIL — `uninitialized constant Traject::SolrPool::Connection`.
|
|
271
|
+
|
|
272
|
+
- [ ] **Step 3: Implement Connection**
|
|
273
|
+
|
|
274
|
+
Create `lib/traject/solr_pool/connection.rb`:
|
|
275
|
+
|
|
276
|
+
```ruby
|
|
277
|
+
# frozen_string_literal: true
|
|
278
|
+
|
|
279
|
+
require 'http_connection_pool'
|
|
280
|
+
|
|
281
|
+
module Traject
|
|
282
|
+
module SolrPool
|
|
283
|
+
# Reusable persistent-connection seam over http_connection_pool. Owns origin
|
|
284
|
+
# binding and pool_options construction; exposes post/get that borrow a
|
|
285
|
+
# pooled HTTP::Session. A future reader can reuse this unchanged.
|
|
286
|
+
class Connection
|
|
287
|
+
attr_reader :origin
|
|
288
|
+
|
|
289
|
+
def initialize(origin:, pool_size:, pool_timeout: nil, headers: {}, auth: nil, timeout: nil)
|
|
290
|
+
@origin = origin
|
|
291
|
+
@pool_size = pool_size
|
|
292
|
+
@pool_timeout = pool_timeout
|
|
293
|
+
@pool_options = build_pool_options(headers, auth, timeout)
|
|
294
|
+
@adapter = build_adapter
|
|
295
|
+
end
|
|
296
|
+
|
|
297
|
+
def post(path, body:)
|
|
298
|
+
@adapter.with_connection { |conn| conn.post(path, body: body) }
|
|
299
|
+
end
|
|
300
|
+
|
|
301
|
+
def get(path, params: {})
|
|
302
|
+
@adapter.with_connection { |conn| conn.get(path, params: params) }
|
|
303
|
+
end
|
|
304
|
+
|
|
305
|
+
def release
|
|
306
|
+
@adapter.release_connection_pool
|
|
307
|
+
end
|
|
308
|
+
|
|
309
|
+
private
|
|
310
|
+
|
|
311
|
+
# Auth is passed through http_connection_pool's :auth option, which folds
|
|
312
|
+
# it into an Authorization header. Nil values are dropped so they never
|
|
313
|
+
# widen the pool key.
|
|
314
|
+
def build_pool_options(headers, auth, timeout)
|
|
315
|
+
opts = { headers: headers }
|
|
316
|
+
opts[:auth] = auth if auth
|
|
317
|
+
opts[:timeout] = timeout if timeout
|
|
318
|
+
opts
|
|
319
|
+
end
|
|
320
|
+
|
|
321
|
+
def build_adapter
|
|
322
|
+
origin = @origin
|
|
323
|
+
pool_size = @pool_size
|
|
324
|
+
pool_timeout = @pool_timeout
|
|
325
|
+
pool_options = @pool_options
|
|
326
|
+
|
|
327
|
+
adapter = Object.new
|
|
328
|
+
adapter.extend(HttpConnectionPool::Connectable)
|
|
329
|
+
adapter.base_url = origin
|
|
330
|
+
adapter.pool_size = pool_size
|
|
331
|
+
adapter.pool_timeout = pool_timeout if pool_timeout
|
|
332
|
+
adapter.pool_options = pool_options
|
|
333
|
+
adapter
|
|
334
|
+
end
|
|
335
|
+
|
|
336
|
+
def pool
|
|
337
|
+
@adapter.connection_pool
|
|
338
|
+
end
|
|
339
|
+
end
|
|
340
|
+
end
|
|
341
|
+
end
|
|
342
|
+
```
|
|
343
|
+
|
|
344
|
+
- [ ] **Step 4: Wire the require into the entry point**
|
|
345
|
+
|
|
346
|
+
Modify `lib/traject/solr_pool.rb` to require the connection (add after the version require):
|
|
347
|
+
|
|
348
|
+
```ruby
|
|
349
|
+
require_relative 'solr_pool/connection'
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
- [ ] **Step 5: Run tests to verify they pass**
|
|
353
|
+
|
|
354
|
+
Run: `bundle exec rspec spec/traject/solr_pool/connection_spec.rb`
|
|
355
|
+
Expected: PASS (5 examples). If http.rb rejects `params: {}` on a plain get, pass `params:` only when non-empty — adjust `get` to `path` vs `path, params:` accordingly and re-run.
|
|
356
|
+
|
|
357
|
+
- [ ] **Step 6: RuboCop**
|
|
358
|
+
|
|
359
|
+
Run: `bundle exec rubocop lib/traject/solr_pool/connection.rb spec/traject/solr_pool/connection_spec.rb`
|
|
360
|
+
Expected: clean (run `bundle exec rubocop -a` to auto-fix, then re-run).
|
|
361
|
+
|
|
362
|
+
- [ ] **Step 7: Commit**
|
|
363
|
+
|
|
364
|
+
```bash
|
|
365
|
+
git add lib/traject/solr_pool/connection.rb lib/traject/solr_pool.rb spec/traject/solr_pool/connection_spec.rb
|
|
366
|
+
git commit -m "Add Connection: origin-bound pooled HTTP seam"
|
|
367
|
+
```
|
|
368
|
+
|
|
369
|
+
---
|
|
370
|
+
|
|
371
|
+
## Task 3: Writer skeleton — settings, URL derivation, Connection wiring
|
|
372
|
+
|
|
373
|
+
**Files:**
|
|
374
|
+
- Create: `lib/traject/solr_pool/solr_json_writer.rb`
|
|
375
|
+
- Create: `spec/traject/solr_pool/solr_json_writer_spec.rb`
|
|
376
|
+
- Modify: `lib/traject/solr_pool.rb`
|
|
377
|
+
|
|
378
|
+
**Interfaces:**
|
|
379
|
+
- Consumes: `Traject::SolrPool::Connection` from Task 2.
|
|
380
|
+
- Produces: `Traject::SolrPool::SolrJsonWriter.new(settings)` responding to `#settings`, `#solr_update_url`, `#connection` (a `Connection`), `#pool_size`; class `Traject::SolrPool::SolrJsonWriter::BadHttpResponse < RuntimeError` with `#response`; `#solr_update_url_with_query(query_params)`.
|
|
381
|
+
|
|
382
|
+
- [ ] **Step 1: Write failing tests for construction and URL derivation**
|
|
383
|
+
|
|
384
|
+
Create `spec/traject/solr_pool/solr_json_writer_spec.rb`:
|
|
385
|
+
|
|
386
|
+
```ruby
|
|
387
|
+
# frozen_string_literal: true
|
|
388
|
+
|
|
389
|
+
RSpec.describe Traject::SolrPool::SolrJsonWriter, :solr_stub do
|
|
390
|
+
def writer(settings = {})
|
|
391
|
+
described_class.new({ 'solr.url' => 'http://solr.test:8983/solr/core' }.merge(settings))
|
|
392
|
+
end
|
|
393
|
+
|
|
394
|
+
describe 'update url derivation' do
|
|
395
|
+
it 'derives the update handler from solr.url' do
|
|
396
|
+
expect(writer.solr_update_url).to eq('http://solr.test:8983/solr/core/update/json')
|
|
397
|
+
end
|
|
398
|
+
|
|
399
|
+
it 'uses solr.update_url verbatim when provided' do
|
|
400
|
+
w = described_class.new('solr.update_url' => 'http://solr.test:8983/solr/core/update')
|
|
401
|
+
expect(w.solr_update_url).to eq('http://solr.test:8983/solr/core/update')
|
|
402
|
+
end
|
|
403
|
+
|
|
404
|
+
it 'raises when neither solr.url nor solr.update_url is set' do
|
|
405
|
+
expect { described_class.new({}) }.to raise_error(ArgumentError)
|
|
406
|
+
end
|
|
407
|
+
end
|
|
408
|
+
|
|
409
|
+
describe 'connection binding' do
|
|
410
|
+
it 'builds a Connection bound to the solr origin' do
|
|
411
|
+
expect(writer.connection.origin).to eq('http://solr.test:8983')
|
|
412
|
+
end
|
|
413
|
+
end
|
|
414
|
+
|
|
415
|
+
describe 'pool size' do
|
|
416
|
+
it 'defaults pool_size to the writer thread pool plus caller headroom' do
|
|
417
|
+
expect(writer('solr_writer.thread_pool' => 3).pool_size).to eq(4)
|
|
418
|
+
end
|
|
419
|
+
|
|
420
|
+
it 'honours an explicit solr_pool.pool_size' do
|
|
421
|
+
expect(writer('solr_pool.pool_size' => 9).pool_size).to eq(9)
|
|
422
|
+
end
|
|
423
|
+
end
|
|
424
|
+
|
|
425
|
+
describe 'query params' do
|
|
426
|
+
it 'appends solr_update_args as a query string' do
|
|
427
|
+
w = writer('solr_writer.solr_update_args' => { commitWithin: 1000 })
|
|
428
|
+
expect(w.solr_update_url_with_query(w.instance_variable_get(:@solr_update_args)))
|
|
429
|
+
.to eq('http://solr.test:8983/solr/core/update/json?commitWithin=1000')
|
|
430
|
+
end
|
|
431
|
+
end
|
|
432
|
+
end
|
|
433
|
+
```
|
|
434
|
+
|
|
435
|
+
- [ ] **Step 2: Run to verify failure**
|
|
436
|
+
|
|
437
|
+
Run: `bundle exec rspec spec/traject/solr_pool/solr_json_writer_spec.rb`
|
|
438
|
+
Expected: FAIL — `uninitialized constant Traject::SolrPool::SolrJsonWriter`.
|
|
439
|
+
|
|
440
|
+
- [ ] **Step 3: Implement the writer skeleton**
|
|
441
|
+
|
|
442
|
+
Create `lib/traject/solr_pool/solr_json_writer.rb`:
|
|
443
|
+
|
|
444
|
+
```ruby
|
|
445
|
+
# frozen_string_literal: true
|
|
446
|
+
|
|
447
|
+
require 'json'
|
|
448
|
+
require 'uri'
|
|
449
|
+
require 'concurrent/atomic/atomic_fixnum'
|
|
450
|
+
require 'traject/util'
|
|
451
|
+
require 'traject/thread_pool'
|
|
452
|
+
|
|
453
|
+
require_relative 'connection'
|
|
454
|
+
|
|
455
|
+
module Traject
|
|
456
|
+
module SolrPool
|
|
457
|
+
# Drop-in replacement for Traject::SolrJsonWriter that sends every Solr HTTP
|
|
458
|
+
# call through a persistent http_connection_pool pool. Reuses the stock
|
|
459
|
+
# writer's settings vocabulary and public surface; all HTTP is delegated to
|
|
460
|
+
# Traject::SolrPool::Connection.
|
|
461
|
+
class SolrJsonWriter
|
|
462
|
+
URI_REGEXP = URI::DEFAULT_PARSER.make_regexp.freeze
|
|
463
|
+
|
|
464
|
+
DEFAULT_MAX_SKIPPED = 0
|
|
465
|
+
DEFAULT_BATCH_SIZE = 100
|
|
466
|
+
|
|
467
|
+
attr_reader :settings, :thread_pool_size, :batched_queue, :solr_update_url,
|
|
468
|
+
:connection, :pool_size
|
|
469
|
+
|
|
470
|
+
def initialize(arg_settings)
|
|
471
|
+
@settings = Traject::Indexer::Settings.new(arg_settings)
|
|
472
|
+
|
|
473
|
+
@max_skipped = (@settings['solr_writer.max_skipped'] || DEFAULT_MAX_SKIPPED).to_i
|
|
474
|
+
@max_skipped = nil if @max_skipped.negative?
|
|
475
|
+
|
|
476
|
+
@solr_update_url, basic_auth_user, basic_auth_password = determine_solr_update_url
|
|
477
|
+
|
|
478
|
+
@batch_size = (@settings['solr_writer.batch_size'] || DEFAULT_BATCH_SIZE).to_i
|
|
479
|
+
@batch_size = 1 if @batch_size < 1
|
|
480
|
+
|
|
481
|
+
@skipped_record_incrementer = Concurrent::AtomicFixnum.new(0)
|
|
482
|
+
@thread_pool_size = (@settings['solr_writer.thread_pool'] || 1).to_i
|
|
483
|
+
|
|
484
|
+
@pool_size = (@settings['solr_pool.pool_size'] || (@thread_pool_size + 1)).to_i
|
|
485
|
+
@connection = build_connection(basic_auth_user, basic_auth_password)
|
|
486
|
+
|
|
487
|
+
@batched_queue = Queue.new
|
|
488
|
+
@thread_pool = Traject::ThreadPool.new(@thread_pool_size)
|
|
489
|
+
|
|
490
|
+
@commit_on_close = (@settings['solr_writer.commit_on_close'] ||
|
|
491
|
+
@settings['solrj_writer.commit_on_close']).to_s == 'true'
|
|
492
|
+
@solr_update_args = @settings['solr_writer.solr_update_args']
|
|
493
|
+
@commit_solr_update_args = @settings['solr_writer.commit_solr_update_args']
|
|
494
|
+
|
|
495
|
+
logger.info(" #{self.class.name} writing to '#{@solr_update_url}' " \
|
|
496
|
+
"#{'(with HTTP basic auth) ' if basic_auth_user || basic_auth_password}" \
|
|
497
|
+
"in batches of #{@batch_size} with #{@thread_pool_size} bg threads")
|
|
498
|
+
end
|
|
499
|
+
|
|
500
|
+
def solr_update_url_with_query(query_params)
|
|
501
|
+
return @solr_update_url unless query_params
|
|
502
|
+
|
|
503
|
+
"#{@solr_update_url}?#{URI.encode_www_form(query_params)}"
|
|
504
|
+
end
|
|
505
|
+
|
|
506
|
+
def logger
|
|
507
|
+
settings['logger'] ||= Yell.new($stderr, level: 'gt.fatal')
|
|
508
|
+
end
|
|
509
|
+
|
|
510
|
+
private
|
|
511
|
+
|
|
512
|
+
# Splits the origin (scheme://host:port -> pool base_url) from the request
|
|
513
|
+
# path, and hands auth to the Connection as an Authorization header value
|
|
514
|
+
# so credentials live in pool_options, never in the origin or logs.
|
|
515
|
+
def build_connection(user, password)
|
|
516
|
+
uri = URI.parse(@solr_update_url)
|
|
517
|
+
origin = "#{uri.scheme}://#{uri.host}:#{uri.port}"
|
|
518
|
+
|
|
519
|
+
Connection.new(
|
|
520
|
+
origin: origin,
|
|
521
|
+
pool_size: @pool_size,
|
|
522
|
+
pool_timeout: @settings['solr_writer.pool_timeout'],
|
|
523
|
+
auth: basic_auth_header(user, password),
|
|
524
|
+
timeout: @settings['solr_writer.http_timeout']
|
|
525
|
+
)
|
|
526
|
+
end
|
|
527
|
+
|
|
528
|
+
def basic_auth_header(user, password)
|
|
529
|
+
return nil unless user || password
|
|
530
|
+
|
|
531
|
+
require 'base64'
|
|
532
|
+
"Basic #{Base64.strict_encode64("#{user}:#{password}")}"
|
|
533
|
+
end
|
|
534
|
+
|
|
535
|
+
def request_path
|
|
536
|
+
uri = URI.parse(@solr_update_url)
|
|
537
|
+
uri.query ? "#{uri.path}?#{uri.query}" : uri.path
|
|
538
|
+
end
|
|
539
|
+
|
|
540
|
+
def determine_solr_update_url
|
|
541
|
+
url = if settings['solr.update_url']
|
|
542
|
+
check_solr_update_url(settings['solr.update_url'])
|
|
543
|
+
else
|
|
544
|
+
derive_solr_update_url_from_solr_url(settings['solr.url'])
|
|
545
|
+
end
|
|
546
|
+
|
|
547
|
+
parsed = URI.parse(url)
|
|
548
|
+
user_from_uri = parsed.user
|
|
549
|
+
password_from_uri = parsed.password
|
|
550
|
+
parsed.user = nil
|
|
551
|
+
parsed.password = nil
|
|
552
|
+
|
|
553
|
+
[parsed.to_s,
|
|
554
|
+
@settings['solr_writer.basic_auth_user'] || user_from_uri,
|
|
555
|
+
@settings['solr_writer.basic_auth_password'] || password_from_uri]
|
|
556
|
+
end
|
|
557
|
+
|
|
558
|
+
def check_solr_update_url(url)
|
|
559
|
+
unless /^#{URI_REGEXP}$/.match?(url)
|
|
560
|
+
raise ArgumentError, "#{self.class.name} setting `solr.update_url` doesn't look like a URL: `#{url}`"
|
|
561
|
+
end
|
|
562
|
+
|
|
563
|
+
url
|
|
564
|
+
end
|
|
565
|
+
|
|
566
|
+
def derive_solr_update_url_from_solr_url(url)
|
|
567
|
+
raise ArgumentError, "#{self.class.name}: Neither solr.update_url nor solr.url set; need at least one" if url.nil?
|
|
568
|
+
|
|
569
|
+
unless /^#{URI_REGEXP}$/.match?(url)
|
|
570
|
+
raise ArgumentError, "#{self.class.name} setting `solr.url` doesn't look like a URL: `#{url}`"
|
|
571
|
+
end
|
|
572
|
+
|
|
573
|
+
[url.chomp('/'), 'update', 'json'].join('/')
|
|
574
|
+
end
|
|
575
|
+
end
|
|
576
|
+
end
|
|
577
|
+
end
|
|
578
|
+
```
|
|
579
|
+
|
|
580
|
+
- [ ] **Step 4: Require yell and wire the entry point**
|
|
581
|
+
|
|
582
|
+
At the top of `solr_json_writer.rb` the writer uses `Yell` and `Traject::Indexer::Settings`; both load via `require 'traject'`. Modify `lib/traject/solr_pool.rb` to require the writer after the connection:
|
|
583
|
+
|
|
584
|
+
```ruby
|
|
585
|
+
require_relative 'solr_pool/solr_json_writer'
|
|
586
|
+
```
|
|
587
|
+
|
|
588
|
+
Confirm `require 'traject'` at the top of `lib/traject/solr_pool.rb` remains (it defines `Traject::Indexer::Settings` and pulls in `yell`).
|
|
589
|
+
|
|
590
|
+
- [ ] **Step 5: Run tests to verify they pass**
|
|
591
|
+
|
|
592
|
+
Run: `bundle exec rspec spec/traject/solr_pool/solr_json_writer_spec.rb`
|
|
593
|
+
Expected: PASS (8 examples).
|
|
594
|
+
|
|
595
|
+
- [ ] **Step 6: RuboCop**
|
|
596
|
+
|
|
597
|
+
Run: `bundle exec rubocop -a lib/traject/solr_pool/solr_json_writer.rb spec/traject/solr_pool/solr_json_writer_spec.rb && bundle exec rubocop lib/traject/solr_pool/solr_json_writer.rb spec/traject/solr_pool/solr_json_writer_spec.rb`
|
|
598
|
+
Expected: clean. If `Metrics/MethodLength` flags `initialize`, extract a private `configure_from_settings` helper rather than disabling the cop.
|
|
599
|
+
|
|
600
|
+
- [ ] **Step 7: Commit**
|
|
601
|
+
|
|
602
|
+
```bash
|
|
603
|
+
git add lib/traject/solr_pool/solr_json_writer.rb lib/traject/solr_pool.rb spec/traject/solr_pool/solr_json_writer_spec.rb
|
|
604
|
+
git commit -m "Add writer skeleton: settings, URL derivation, Connection wiring"
|
|
605
|
+
```
|
|
606
|
+
|
|
607
|
+
---
|
|
608
|
+
|
|
609
|
+
## Task 4: Writer output path — put, send_batch, send_single, BadHttpResponse
|
|
610
|
+
|
|
611
|
+
**Files:**
|
|
612
|
+
- Modify: `lib/traject/solr_pool/solr_json_writer.rb`
|
|
613
|
+
- Modify: `spec/traject/solr_pool/solr_json_writer_spec.rb`
|
|
614
|
+
|
|
615
|
+
**Interfaces:**
|
|
616
|
+
- Consumes: `#connection`, `#request_path`, `@batched_queue`, `@thread_pool` from Task 3.
|
|
617
|
+
- Produces: `#put(context)`, `#flush`, `#send_batch(batch)`, `#send_single(context)`, `#skipped_record_count`; nested `BadHttpResponse < RuntimeError` (with `#response`) and `MaxSkippedRecordsExceeded < RuntimeError`; default `#skippable_exceptions`.
|
|
618
|
+
|
|
619
|
+
- [ ] **Step 1: Write failing tests for the output path**
|
|
620
|
+
|
|
621
|
+
Add to `spec/traject/solr_pool/solr_json_writer_spec.rb` inside the top-level describe:
|
|
622
|
+
|
|
623
|
+
```ruby
|
|
624
|
+
def context(hash)
|
|
625
|
+
Traject::Indexer::Context.new(output_hash: hash)
|
|
626
|
+
end
|
|
627
|
+
|
|
628
|
+
describe 'put and batching' do
|
|
629
|
+
it 'posts a full batch as a JSON array once batch_size is reached' do
|
|
630
|
+
stub = stub_solr_update('http://solr.test:8983/solr/core/update/json')
|
|
631
|
+
w = writer('solr_writer.batch_size' => 2, 'solr_writer.thread_pool' => 0)
|
|
632
|
+
w.put(context('id' => '1'))
|
|
633
|
+
w.put(context('id' => '2'))
|
|
634
|
+
expect(stub.with { |req| JSON.parse(req.body).length == 2 }).to have_been_requested
|
|
635
|
+
end
|
|
636
|
+
|
|
637
|
+
it 'does not post before the batch is full' do
|
|
638
|
+
stub = stub_solr_update('http://solr.test:8983/solr/core/update/json')
|
|
639
|
+
w = writer('solr_writer.batch_size' => 5, 'solr_writer.thread_pool' => 0)
|
|
640
|
+
w.put(context('id' => '1'))
|
|
641
|
+
expect(stub).not_to have_been_requested
|
|
642
|
+
end
|
|
643
|
+
end
|
|
644
|
+
|
|
645
|
+
describe 'error handling' do
|
|
646
|
+
it 'retries a failed batch as individual records' do
|
|
647
|
+
stub_request(:post, 'http://solr.test:8983/solr/core/update/json')
|
|
648
|
+
.to_return({ status: 500, body: '{}' }, { status: 200, body: '{}' })
|
|
649
|
+
w = writer('solr_writer.batch_size' => 1, 'solr_writer.thread_pool' => 0,
|
|
650
|
+
'solr_writer.max_skipped' => -1)
|
|
651
|
+
expect { w.put(context('id' => '1')) }.not_to raise_error
|
|
652
|
+
end
|
|
653
|
+
|
|
654
|
+
it 'raises MaxSkippedRecordsExceeded past the skip limit' do
|
|
655
|
+
stub_solr_update('http://solr.test:8983/solr/core/update/json', status: 500, body: '{}')
|
|
656
|
+
w = writer('solr_writer.batch_size' => 1, 'solr_writer.thread_pool' => 0,
|
|
657
|
+
'solr_writer.max_skipped' => 0)
|
|
658
|
+
expect { w.put(context('id' => '1')) }
|
|
659
|
+
.to raise_error(described_class::MaxSkippedRecordsExceeded)
|
|
660
|
+
end
|
|
661
|
+
|
|
662
|
+
it 'counts skipped records' do
|
|
663
|
+
stub_solr_update('http://solr.test:8983/solr/core/update/json', status: 500, body: '{}')
|
|
664
|
+
w = writer('solr_writer.batch_size' => 1, 'solr_writer.thread_pool' => 0,
|
|
665
|
+
'solr_writer.max_skipped' => -1)
|
|
666
|
+
w.put(context('id' => '1'))
|
|
667
|
+
expect(w.skipped_record_count).to eq(1)
|
|
668
|
+
end
|
|
669
|
+
end
|
|
670
|
+
```
|
|
671
|
+
|
|
672
|
+
- [ ] **Step 2: Run to verify failure**
|
|
673
|
+
|
|
674
|
+
Run: `bundle exec rspec spec/traject/solr_pool/solr_json_writer_spec.rb -e 'put and batching'`
|
|
675
|
+
Expected: FAIL — `undefined method 'put'`.
|
|
676
|
+
|
|
677
|
+
- [ ] **Step 3: Implement the output path**
|
|
678
|
+
|
|
679
|
+
Add these public methods to `SolrJsonWriter` (above `private`), and the nested classes + `skippable_exceptions` at the appropriate scope:
|
|
680
|
+
|
|
681
|
+
```ruby
|
|
682
|
+
def put(context)
|
|
683
|
+
@thread_pool.raise_collected_exception!
|
|
684
|
+
@batched_queue << context
|
|
685
|
+
return if @batched_queue.size < @batch_size
|
|
686
|
+
|
|
687
|
+
batch = Traject::Util.drain_queue(@batched_queue)
|
|
688
|
+
@thread_pool.maybe_in_thread_pool(batch) { |b| send_batch(b) }
|
|
689
|
+
end
|
|
690
|
+
|
|
691
|
+
def flush
|
|
692
|
+
send_batch(Traject::Util.drain_queue(@batched_queue))
|
|
693
|
+
end
|
|
694
|
+
|
|
695
|
+
def send_batch(batch)
|
|
696
|
+
return if batch.empty?
|
|
697
|
+
|
|
698
|
+
json_package = JSON.generate(batch.map(&:output_hash))
|
|
699
|
+
begin
|
|
700
|
+
resp = connection.post(request_path, body: json_package)
|
|
701
|
+
rescue StandardError => e
|
|
702
|
+
exception = e
|
|
703
|
+
end
|
|
704
|
+
|
|
705
|
+
return if exception.nil? && resp.status == 200
|
|
706
|
+
|
|
707
|
+
log_batch_failure(exception, resp)
|
|
708
|
+
batch.each { |c| send_single(c) }
|
|
709
|
+
end
|
|
710
|
+
|
|
711
|
+
def send_single(context)
|
|
712
|
+
json_package = JSON.generate([context.output_hash])
|
|
713
|
+
resp = connection.post(request_path, body: json_package)
|
|
714
|
+
unless resp.status == 200
|
|
715
|
+
raise BadHttpResponse.new("Unexpected HTTP response status #{resp.code} from POST", resp)
|
|
716
|
+
end
|
|
717
|
+
rescue *skippable_exceptions => e
|
|
718
|
+
handle_skipped(context, e)
|
|
719
|
+
end
|
|
720
|
+
|
|
721
|
+
def skipped_record_count
|
|
722
|
+
@skipped_record_incrementer.value
|
|
723
|
+
end
|
|
724
|
+
```
|
|
725
|
+
|
|
726
|
+
Add these private helpers:
|
|
727
|
+
|
|
728
|
+
```ruby
|
|
729
|
+
def log_batch_failure(exception, resp)
|
|
730
|
+
message = if exception
|
|
731
|
+
Traject::Util.exception_to_log_message(exception)
|
|
732
|
+
else
|
|
733
|
+
"Solr response: #{resp.code}: #{resp.to_s}"
|
|
734
|
+
end
|
|
735
|
+
logger.error("Error in Solr batch add. Will retry documents individually " \
|
|
736
|
+
"at performance penalty: #{message}")
|
|
737
|
+
end
|
|
738
|
+
|
|
739
|
+
def handle_skipped(context, exception)
|
|
740
|
+
msg = if exception.is_a?(BadHttpResponse)
|
|
741
|
+
"Solr error response: #{exception.response.code}: #{exception.response.to_s}"
|
|
742
|
+
else
|
|
743
|
+
Traject::Util.exception_to_log_message(exception)
|
|
744
|
+
end
|
|
745
|
+
logger.error("Could not add record #{context.record_inspect}: #{msg}")
|
|
746
|
+
|
|
747
|
+
@skipped_record_incrementer.increment
|
|
748
|
+
return unless @max_skipped && skipped_record_count > @max_skipped
|
|
749
|
+
|
|
750
|
+
raise MaxSkippedRecordsExceeded,
|
|
751
|
+
"#{self.class.name}: Exceeded maximum number of skipped records " \
|
|
752
|
+
"(#{@max_skipped}): aborting: #{exception.message}"
|
|
753
|
+
end
|
|
754
|
+
|
|
755
|
+
def skippable_exceptions
|
|
756
|
+
@skippable_exceptions ||= settings['solr_writer.skippable_exceptions'] || [
|
|
757
|
+
HTTP::TimeoutError,
|
|
758
|
+
HttpConnectionPool::TimeoutError,
|
|
759
|
+
HTTP::ConnectionError,
|
|
760
|
+
SocketError,
|
|
761
|
+
Errno::ECONNREFUSED,
|
|
762
|
+
BadHttpResponse
|
|
763
|
+
]
|
|
764
|
+
end
|
|
765
|
+
```
|
|
766
|
+
|
|
767
|
+
Add nested classes inside the `SolrJsonWriter` class body (near the top, after the constants):
|
|
768
|
+
|
|
769
|
+
```ruby
|
|
770
|
+
class MaxSkippedRecordsExceeded < RuntimeError; end
|
|
771
|
+
|
|
772
|
+
# Mirrors the stock writer's BadHttpResponse, but its #response is an
|
|
773
|
+
# http.rb HTTP::Response (use #code / #to_s), not an HTTPClient message.
|
|
774
|
+
class BadHttpResponse < RuntimeError
|
|
775
|
+
attr_reader :response
|
|
776
|
+
|
|
777
|
+
def initialize(msg, response = nil)
|
|
778
|
+
solr_error = find_solr_error(response)
|
|
779
|
+
msg = "#{msg}: #{solr_error}" if solr_error
|
|
780
|
+
super(msg)
|
|
781
|
+
@response = response
|
|
782
|
+
end
|
|
783
|
+
|
|
784
|
+
private
|
|
785
|
+
|
|
786
|
+
def find_solr_error(response)
|
|
787
|
+
return nil unless response&.to_s && response.headers['Content-Type'].to_s.start_with?('application/json')
|
|
788
|
+
|
|
789
|
+
JSON.parse(response.to_s).dig('error', 'msg')
|
|
790
|
+
rescue JSON::ParserError
|
|
791
|
+
nil
|
|
792
|
+
end
|
|
793
|
+
end
|
|
794
|
+
```
|
|
795
|
+
|
|
796
|
+
Ensure `require 'http'` is present at the top of the file (for the error constants). Add it with the other requires.
|
|
797
|
+
|
|
798
|
+
- [ ] **Step 4: Run tests to verify they pass**
|
|
799
|
+
|
|
800
|
+
Run: `bundle exec rspec spec/traject/solr_pool/solr_json_writer_spec.rb`
|
|
801
|
+
Expected: PASS (all examples). Note `solr_writer.thread_pool => 0` runs inline (null thread pool), so failures surface synchronously.
|
|
802
|
+
|
|
803
|
+
- [ ] **Step 5: RuboCop**
|
|
804
|
+
|
|
805
|
+
Run: `bundle exec rubocop -a lib/traject/solr_pool/solr_json_writer.rb spec/traject/solr_pool/solr_json_writer_spec.rb && bundle exec rubocop lib/traject/solr_pool/solr_json_writer.rb spec/traject/solr_pool/solr_json_writer_spec.rb`
|
|
806
|
+
Expected: clean. `Style/StringConcatenation` / `Lint/RedundantStringCoercion` may flag `resp.to_s` in interpolation — replace `"#{...}: #{resp.to_s}"` with `"#{...}: #{resp}"` where the cop asks.
|
|
807
|
+
|
|
808
|
+
- [ ] **Step 6: Commit**
|
|
809
|
+
|
|
810
|
+
```bash
|
|
811
|
+
git add lib/traject/solr_pool/solr_json_writer.rb spec/traject/solr_pool/solr_json_writer_spec.rb
|
|
812
|
+
git commit -m "Add writer output path: put/send_batch/send_single with error mapping"
|
|
813
|
+
```
|
|
814
|
+
|
|
815
|
+
---
|
|
816
|
+
|
|
817
|
+
## Task 5: Writer lifecycle — close, commit, delete, delete_all!
|
|
818
|
+
|
|
819
|
+
**Files:**
|
|
820
|
+
- Modify: `lib/traject/solr_pool/solr_json_writer.rb`
|
|
821
|
+
- Modify: `spec/traject/solr_pool/solr_json_writer_spec.rb`
|
|
822
|
+
|
|
823
|
+
**Interfaces:**
|
|
824
|
+
- Consumes: everything from Tasks 3–4.
|
|
825
|
+
- Produces: `#close`, `#commit(query_params = nil)`, `#delete(id)`, `#delete_all!`. `#close` flushes, drains the thread pool, optionally commits, and does NOT release the pool (leaves it warm).
|
|
826
|
+
|
|
827
|
+
- [ ] **Step 1: Write failing tests for lifecycle**
|
|
828
|
+
|
|
829
|
+
Add to the spec file:
|
|
830
|
+
|
|
831
|
+
```ruby
|
|
832
|
+
describe 'close' do
|
|
833
|
+
it 'flushes queued records on close' do
|
|
834
|
+
stub = stub_solr_update('http://solr.test:8983/solr/core/update/json')
|
|
835
|
+
w = writer('solr_writer.batch_size' => 100, 'solr_writer.thread_pool' => 0)
|
|
836
|
+
w.put(context('id' => '1'))
|
|
837
|
+
w.close
|
|
838
|
+
expect(stub).to have_been_requested
|
|
839
|
+
end
|
|
840
|
+
|
|
841
|
+
it 'leaves the pool warm after close' do
|
|
842
|
+
stub_solr_update('http://solr.test:8983/solr/core/update/json')
|
|
843
|
+
w = writer('solr_writer.thread_pool' => 0)
|
|
844
|
+
origin = w.connection.origin
|
|
845
|
+
w.close
|
|
846
|
+
expect(HttpConnectionPool::Registry.instance.stats.map { |s| s[:origin] }).to include(origin)
|
|
847
|
+
end
|
|
848
|
+
|
|
849
|
+
it 'commits on close when configured' do
|
|
850
|
+
stub_solr_update('http://solr.test:8983/solr/core/update/json')
|
|
851
|
+
commit = stub_solr_get('http://solr.test:8983/solr/core/update/json?commit=true')
|
|
852
|
+
w = writer('solr_writer.thread_pool' => 0, 'solr_writer.commit_on_close' => 'true')
|
|
853
|
+
w.close
|
|
854
|
+
expect(commit).to have_been_requested
|
|
855
|
+
end
|
|
856
|
+
end
|
|
857
|
+
|
|
858
|
+
describe 'delete' do
|
|
859
|
+
it 'posts a delete-by-id document' do
|
|
860
|
+
stub = stub_solr_update('http://solr.test:8983/solr/core/update/json')
|
|
861
|
+
.with { |req| JSON.parse(req.body) == { 'delete' => 'abc' } }
|
|
862
|
+
writer('solr_writer.thread_pool' => 0).delete('abc')
|
|
863
|
+
expect(stub).to have_been_requested
|
|
864
|
+
end
|
|
865
|
+
|
|
866
|
+
it 'raises on a non-200 delete' do
|
|
867
|
+
stub_solr_update('http://solr.test:8983/solr/core/update/json', status: 500)
|
|
868
|
+
expect { writer('solr_writer.thread_pool' => 0).delete('abc') }.to raise_error(RuntimeError)
|
|
869
|
+
end
|
|
870
|
+
end
|
|
871
|
+
```
|
|
872
|
+
|
|
873
|
+
- [ ] **Step 2: Run to verify failure**
|
|
874
|
+
|
|
875
|
+
Run: `bundle exec rspec spec/traject/solr_pool/solr_json_writer_spec.rb -e 'close'`
|
|
876
|
+
Expected: FAIL — `undefined method 'close'`.
|
|
877
|
+
|
|
878
|
+
- [ ] **Step 3: Implement lifecycle methods**
|
|
879
|
+
|
|
880
|
+
Add as public methods:
|
|
881
|
+
|
|
882
|
+
```ruby
|
|
883
|
+
def close
|
|
884
|
+
@thread_pool.raise_collected_exception!
|
|
885
|
+
|
|
886
|
+
batch = Traject::Util.drain_queue(@batched_queue)
|
|
887
|
+
@thread_pool.maybe_in_thread_pool { send_batch(batch) } unless batch.empty?
|
|
888
|
+
|
|
889
|
+
if @thread_pool_size&.positive?
|
|
890
|
+
elapsed = @thread_pool.shutdown_and_wait
|
|
891
|
+
logger.warn("Waited #{elapsed}s for all threads") if elapsed > 60
|
|
892
|
+
logger.warn("#{self.class.name}: #{skipped_record_count} skipped records") if skipped_record_count.positive?
|
|
893
|
+
end
|
|
894
|
+
|
|
895
|
+
@thread_pool.raise_collected_exception!
|
|
896
|
+
commit if @commit_on_close
|
|
897
|
+
# Pool is intentionally left warm in the registry for reuse by later
|
|
898
|
+
# writers/readers on this origin; teardown is the host app's concern.
|
|
899
|
+
end
|
|
900
|
+
|
|
901
|
+
def commit(query_params = nil)
|
|
902
|
+
query_params ||= @commit_solr_update_args || { 'commit' => 'true' }
|
|
903
|
+
logger.info("#{self.class.name} sending commit to solr at url #{@solr_update_url}...")
|
|
904
|
+
|
|
905
|
+
resp = connection.get(request_path_for(query_params))
|
|
906
|
+
raise "Could not commit to Solr: #{resp.code} #{resp}" unless resp.status == 200
|
|
907
|
+
end
|
|
908
|
+
|
|
909
|
+
def delete(id)
|
|
910
|
+
json_package = JSON.generate(delete: id)
|
|
911
|
+
resp = connection.post(request_path, body: json_package)
|
|
912
|
+
raise "Could not delete #{id.inspect}, http response #{resp.code}: #{resp}" unless resp.status == 200
|
|
913
|
+
end
|
|
914
|
+
|
|
915
|
+
def delete_all!
|
|
916
|
+
delete(query: '*:*')
|
|
917
|
+
end
|
|
918
|
+
```
|
|
919
|
+
|
|
920
|
+
Add a private helper to build a path with query params for GET-based commit:
|
|
921
|
+
|
|
922
|
+
```ruby
|
|
923
|
+
def request_path_for(query_params)
|
|
924
|
+
base = request_path
|
|
925
|
+
return base unless query_params
|
|
926
|
+
|
|
927
|
+
separator = base.include?('?') ? '&' : '?'
|
|
928
|
+
"#{base}#{separator}#{URI.encode_www_form(query_params)}"
|
|
929
|
+
end
|
|
930
|
+
```
|
|
931
|
+
|
|
932
|
+
Note: the `delete(query: '*:*')` path passes a hash whose value is a query, matching the stock writer, which posts `{delete: id}`; here `id` may be a hash. This mirrors the stock behaviour exactly.
|
|
933
|
+
|
|
934
|
+
- [ ] **Step 4: Run tests to verify they pass**
|
|
935
|
+
|
|
936
|
+
Run: `bundle exec rspec spec/traject/solr_pool/solr_json_writer_spec.rb`
|
|
937
|
+
Expected: PASS (all). If the commit stub URL mismatches, print the actual requested URL with `WebMock.after_request` or check `request_path_for` output against the stub in Step 1 and align.
|
|
938
|
+
|
|
939
|
+
- [ ] **Step 5: RuboCop**
|
|
940
|
+
|
|
941
|
+
Run: `bundle exec rubocop -a lib/traject/solr_pool/solr_json_writer.rb spec/traject/solr_pool/solr_json_writer_spec.rb && bundle exec rubocop lib/traject/solr_pool/solr_json_writer.rb spec/traject/solr_pool/solr_json_writer_spec.rb`
|
|
942
|
+
Expected: clean.
|
|
943
|
+
|
|
944
|
+
- [ ] **Step 6: Commit**
|
|
945
|
+
|
|
946
|
+
```bash
|
|
947
|
+
git add lib/traject/solr_pool/solr_json_writer.rb spec/traject/solr_pool/solr_json_writer_spec.rb
|
|
948
|
+
git commit -m "Add writer lifecycle: close (pool stays warm), commit, delete, delete_all!"
|
|
949
|
+
```
|
|
950
|
+
|
|
951
|
+
---
|
|
952
|
+
|
|
953
|
+
## Task 6: Concurrency integration spec
|
|
954
|
+
|
|
955
|
+
**Files:**
|
|
956
|
+
- Create: `spec/support/thread_safety_helpers.rb`
|
|
957
|
+
- Create: `spec/integration/concurrency_spec.rb`
|
|
958
|
+
- Modify: `spec/spec_helper.rb`
|
|
959
|
+
|
|
960
|
+
**Interfaces:**
|
|
961
|
+
- Consumes: full writer.
|
|
962
|
+
- Produces: `ThreadSafetyHelpers#cyclic_barrier(count)` (tag `:thread_safety`); an integration spec proving no records are lost under a threaded pool.
|
|
963
|
+
|
|
964
|
+
- [ ] **Step 1: Write the thread-safety helper**
|
|
965
|
+
|
|
966
|
+
Create `spec/support/thread_safety_helpers.rb`:
|
|
967
|
+
|
|
968
|
+
```ruby
|
|
969
|
+
# frozen_string_literal: true
|
|
970
|
+
|
|
971
|
+
# Concurrency helpers included into example groups tagged :thread_safety.
|
|
972
|
+
module ThreadSafetyHelpers
|
|
973
|
+
# Returns a lambda that blocks each caller until `count` callers have arrived,
|
|
974
|
+
# so threads start their real work at the same moment.
|
|
975
|
+
def cyclic_barrier(count)
|
|
976
|
+
mutex = Mutex.new
|
|
977
|
+
cond = ConditionVariable.new
|
|
978
|
+
arrived = 0
|
|
979
|
+
lambda do
|
|
980
|
+
mutex.synchronize do
|
|
981
|
+
arrived += 1
|
|
982
|
+
arrived >= count ? cond.broadcast : cond.wait(mutex)
|
|
983
|
+
end
|
|
984
|
+
end
|
|
985
|
+
end
|
|
986
|
+
end
|
|
987
|
+
```
|
|
988
|
+
|
|
989
|
+
- [ ] **Step 2: Wire the helper into spec_helper**
|
|
990
|
+
|
|
991
|
+
Add to `spec/spec_helper.rb` inside `RSpec.configure`:
|
|
992
|
+
|
|
993
|
+
```ruby
|
|
994
|
+
config.include ThreadSafetyHelpers, :thread_safety
|
|
995
|
+
```
|
|
996
|
+
|
|
997
|
+
- [ ] **Step 3: Write the failing concurrency spec**
|
|
998
|
+
|
|
999
|
+
Create `spec/integration/concurrency_spec.rb`:
|
|
1000
|
+
|
|
1001
|
+
```ruby
|
|
1002
|
+
# frozen_string_literal: true
|
|
1003
|
+
|
|
1004
|
+
require 'spec_helper'
|
|
1005
|
+
|
|
1006
|
+
RSpec.describe 'Writer concurrency', :integration, :solr_stub, :thread_safety do
|
|
1007
|
+
it 'delivers every record with a multi-thread writer pool' do
|
|
1008
|
+
received = Concurrent::AtomicFixnum.new(0)
|
|
1009
|
+
stub_request(:post, 'http://solr.test:8983/solr/core/update/json')
|
|
1010
|
+
.to_return do |req|
|
|
1011
|
+
received.increment(JSON.parse(req.body).length)
|
|
1012
|
+
{ status: 200, body: '{}' }
|
|
1013
|
+
end
|
|
1014
|
+
|
|
1015
|
+
writer = Traject::SolrPool::SolrJsonWriter.new(
|
|
1016
|
+
'solr.url' => 'http://solr.test:8983/solr/core',
|
|
1017
|
+
'solr_writer.batch_size' => 10,
|
|
1018
|
+
'solr_writer.thread_pool' => 4,
|
|
1019
|
+
'solr_pool.pool_size' => 4
|
|
1020
|
+
)
|
|
1021
|
+
|
|
1022
|
+
200.times { |i| writer.put(Traject::Indexer::Context.new(output_hash: { 'id' => i.to_s })) }
|
|
1023
|
+
writer.close
|
|
1024
|
+
|
|
1025
|
+
expect(received.value).to eq(200)
|
|
1026
|
+
end
|
|
1027
|
+
end
|
|
1028
|
+
```
|
|
1029
|
+
|
|
1030
|
+
- [ ] **Step 4: Run to verify it passes (behaviour already implemented)**
|
|
1031
|
+
|
|
1032
|
+
Run: `bundle exec rspec spec/integration/concurrency_spec.rb`
|
|
1033
|
+
Expected: PASS. (This is an integration guard; the writer already implements the behaviour, so it should pass once wiring is correct. If it fails on lost records, that is a real thread-safety bug — STOP and debug with superpowers:systematic-debugging, do not weaken the assertion.)
|
|
1034
|
+
|
|
1035
|
+
- [ ] **Step 5: RuboCop**
|
|
1036
|
+
|
|
1037
|
+
Run: `bundle exec rubocop -a spec/integration/concurrency_spec.rb spec/support/thread_safety_helpers.rb && bundle exec rubocop spec/integration/concurrency_spec.rb spec/support/thread_safety_helpers.rb spec/spec_helper.rb`
|
|
1038
|
+
Expected: clean.
|
|
1039
|
+
|
|
1040
|
+
- [ ] **Step 6: Commit**
|
|
1041
|
+
|
|
1042
|
+
```bash
|
|
1043
|
+
git add spec/support/thread_safety_helpers.rb spec/integration/concurrency_spec.rb spec/spec_helper.rb
|
|
1044
|
+
git commit -m "Add concurrency integration spec proving no lost records under threaded pool"
|
|
1045
|
+
```
|
|
1046
|
+
|
|
1047
|
+
---
|
|
1048
|
+
|
|
1049
|
+
## Task 7: Background-job (Sidekiq >= 8 / Active Job) integration spec
|
|
1050
|
+
|
|
1051
|
+
**Files:**
|
|
1052
|
+
- Create: `spec/support/job_helpers.rb`
|
|
1053
|
+
- Create: `spec/integration/background_job_spec.rb`
|
|
1054
|
+
- Modify: `spec/spec_helper.rb`
|
|
1055
|
+
|
|
1056
|
+
**Interfaces:**
|
|
1057
|
+
- Consumes: full writer; `sidekiq`, `active_job` (test-only).
|
|
1058
|
+
- Produces: `JobHelpers` (tag `:background_jobs`) with a job that runs the writer; a spec proving the writer works inside a Sidekiq >= 8 worker and shares one pool across jobs.
|
|
1059
|
+
|
|
1060
|
+
- [ ] **Step 1: Write the job helper**
|
|
1061
|
+
|
|
1062
|
+
Create `spec/support/job_helpers.rb`:
|
|
1063
|
+
|
|
1064
|
+
```ruby
|
|
1065
|
+
# frozen_string_literal: true
|
|
1066
|
+
|
|
1067
|
+
require 'sidekiq'
|
|
1068
|
+
require 'sidekiq/testing'
|
|
1069
|
+
|
|
1070
|
+
# Helpers and job classes for the background-job integration spec. Included into
|
|
1071
|
+
# any example group tagged :background_jobs. Sidekiq runs inline (enqueue =
|
|
1072
|
+
# synchronous execution) so we exercise the worker path without Redis.
|
|
1073
|
+
module JobHelpers
|
|
1074
|
+
def self.included(base)
|
|
1075
|
+
base.before { Sidekiq::Testing.inline! }
|
|
1076
|
+
base.after { Sidekiq::Testing.fake! }
|
|
1077
|
+
end
|
|
1078
|
+
|
|
1079
|
+
SOLR_URL = 'http://solr.jobs.test:8983/solr/core'
|
|
1080
|
+
|
|
1081
|
+
class IndexJob
|
|
1082
|
+
include Sidekiq::Job
|
|
1083
|
+
|
|
1084
|
+
def perform(id)
|
|
1085
|
+
writer = Traject::SolrPool::SolrJsonWriter.new(
|
|
1086
|
+
'solr.url' => SOLR_URL,
|
|
1087
|
+
'solr_writer.thread_pool' => 0,
|
|
1088
|
+
'solr_writer.batch_size' => 1
|
|
1089
|
+
)
|
|
1090
|
+
writer.put(Traject::Indexer::Context.new(output_hash: { 'id' => id }))
|
|
1091
|
+
writer.close
|
|
1092
|
+
end
|
|
1093
|
+
end
|
|
1094
|
+
|
|
1095
|
+
def registry
|
|
1096
|
+
HttpConnectionPool::Registry.instance
|
|
1097
|
+
end
|
|
1098
|
+
end
|
|
1099
|
+
```
|
|
1100
|
+
|
|
1101
|
+
- [ ] **Step 2: Wire into spec_helper**
|
|
1102
|
+
|
|
1103
|
+
Add to `spec/spec_helper.rb`:
|
|
1104
|
+
|
|
1105
|
+
```ruby
|
|
1106
|
+
config.include JobHelpers, :background_jobs
|
|
1107
|
+
```
|
|
1108
|
+
|
|
1109
|
+
- [ ] **Step 3: Write the failing spec**
|
|
1110
|
+
|
|
1111
|
+
Create `spec/integration/background_job_spec.rb`:
|
|
1112
|
+
|
|
1113
|
+
```ruby
|
|
1114
|
+
# frozen_string_literal: true
|
|
1115
|
+
|
|
1116
|
+
require 'spec_helper'
|
|
1117
|
+
|
|
1118
|
+
RSpec.describe 'Background job integration', :integration, :background_jobs, :solr_stub do
|
|
1119
|
+
before do
|
|
1120
|
+
stub_request(:post, 'http://solr.jobs.test:8983/solr/core/update/json')
|
|
1121
|
+
.to_return(status: 200, body: '{}')
|
|
1122
|
+
end
|
|
1123
|
+
|
|
1124
|
+
it 'runs the pooled writer inside a Sidekiq job' do
|
|
1125
|
+
JobHelpers::IndexJob.perform_async('doc-1')
|
|
1126
|
+
expect(
|
|
1127
|
+
a_request(:post, 'http://solr.jobs.test:8983/solr/core/update/json')
|
|
1128
|
+
).to have_been_made
|
|
1129
|
+
end
|
|
1130
|
+
|
|
1131
|
+
it 'shares one pool across many sequential jobs against the same origin' do
|
|
1132
|
+
25.times { |i| JobHelpers::IndexJob.perform_async("doc-#{i}") }
|
|
1133
|
+
origins = registry.stats.map { |s| s[:origin] }
|
|
1134
|
+
expect(origins.count { |o| o == 'http://solr.jobs.test:8983' }).to eq(1)
|
|
1135
|
+
end
|
|
1136
|
+
end
|
|
1137
|
+
```
|
|
1138
|
+
|
|
1139
|
+
- [ ] **Step 4: Run to verify it passes**
|
|
1140
|
+
|
|
1141
|
+
Run: `bundle exec rspec spec/integration/background_job_spec.rb`
|
|
1142
|
+
Expected: PASS (2 examples). If Sidekiq 8 requires a Redis client even in inline mode at load, set `Sidekiq.configure_client`/`configure_server` no-ops in the helper or rescue the connection at load; keep the writer behaviour assertions intact.
|
|
1143
|
+
|
|
1144
|
+
- [ ] **Step 5: RuboCop**
|
|
1145
|
+
|
|
1146
|
+
Run: `bundle exec rubocop -a spec/support/job_helpers.rb spec/integration/background_job_spec.rb && bundle exec rubocop spec/support/job_helpers.rb spec/integration/background_job_spec.rb spec/spec_helper.rb`
|
|
1147
|
+
Expected: clean.
|
|
1148
|
+
|
|
1149
|
+
- [ ] **Step 6: Commit**
|
|
1150
|
+
|
|
1151
|
+
```bash
|
|
1152
|
+
git add spec/support/job_helpers.rb spec/integration/background_job_spec.rb spec/spec_helper.rb
|
|
1153
|
+
git commit -m "Add Sidekiq >= 8 background-job integration spec"
|
|
1154
|
+
```
|
|
1155
|
+
|
|
1156
|
+
---
|
|
1157
|
+
|
|
1158
|
+
## Task 8: Zeitwerk and Rails compatibility integration specs
|
|
1159
|
+
|
|
1160
|
+
**Files:**
|
|
1161
|
+
- Create: `spec/integration/zeitwerk_compliance_spec.rb`
|
|
1162
|
+
- Create: `spec/integration/rails_compatibility_spec.rb`
|
|
1163
|
+
- Modify: `.rubocop.yml` (exclude `RSpec/DescribeClass` for integration specs)
|
|
1164
|
+
|
|
1165
|
+
**Interfaces:**
|
|
1166
|
+
- Consumes: the gem's file/constant layout; `zeitwerk`, `activesupport` (test-only).
|
|
1167
|
+
- Produces: proof the layout eager-loads under Zeitwerk in a clean subprocess and coexists with activesupport.
|
|
1168
|
+
|
|
1169
|
+
- [ ] **Step 1: Exclude RSpec/DescribeClass for integration specs**
|
|
1170
|
+
|
|
1171
|
+
Add to `.rubocop.yml`:
|
|
1172
|
+
|
|
1173
|
+
```yaml
|
|
1174
|
+
RSpec/DescribeClass:
|
|
1175
|
+
Exclude:
|
|
1176
|
+
- 'spec/integration/**/*'
|
|
1177
|
+
```
|
|
1178
|
+
|
|
1179
|
+
- [ ] **Step 2: Write the Zeitwerk compliance spec**
|
|
1180
|
+
|
|
1181
|
+
Create `spec/integration/zeitwerk_compliance_spec.rb`. Because the gem lives under `lib/traject/` but only owns the `traject/solr_pool` subtree (traject itself owns `traject/`), the loader pushes `lib` and ignores everything except our files by ignoring the traject entry points. Simplest correct approach: push a loader rooted at `lib`, ignore `traject/solr_pool.rb` (entry) and `traject/solr_pool/version.rb`, and only assert our constants resolve.
|
|
1182
|
+
|
|
1183
|
+
```ruby
|
|
1184
|
+
# frozen_string_literal: true
|
|
1185
|
+
|
|
1186
|
+
require 'spec_helper'
|
|
1187
|
+
|
|
1188
|
+
# Verifies our file/constant layout conforms to Zeitwerk naming so a host Rails
|
|
1189
|
+
# app eager-loading in production never trips over it. The gem itself loads via
|
|
1190
|
+
# plain require (traject convention) and takes no runtime Zeitwerk dependency.
|
|
1191
|
+
#
|
|
1192
|
+
# Subprocess: spec_helper already required the gem, so an in-process loader
|
|
1193
|
+
# would no-op. A clean process exercises a real autoload.
|
|
1194
|
+
RSpec.describe 'Zeitwerk compliance', :integration do
|
|
1195
|
+
let(:gem_root) { File.expand_path('../..', __dir__) }
|
|
1196
|
+
|
|
1197
|
+
let(:probe) do
|
|
1198
|
+
<<~RUBY
|
|
1199
|
+
require 'zeitwerk'
|
|
1200
|
+
lib = File.join(#{gem_root.inspect}, 'lib')
|
|
1201
|
+
loader = Zeitwerk::Loader.new
|
|
1202
|
+
loader.push_dir(lib)
|
|
1203
|
+
# traject/ is owned by the traject gem; we only manage our subtree. Ignore
|
|
1204
|
+
# the entry file and version.rb (defines VERSION, not Version).
|
|
1205
|
+
loader.ignore(File.join(lib, 'traject', 'solr_pool.rb'))
|
|
1206
|
+
loader.ignore(File.join(lib, 'traject', 'solr_pool', 'version.rb'))
|
|
1207
|
+
# Everything else under lib/traject that is not ours must be ignored too;
|
|
1208
|
+
# our gem ships only the solr_pool subtree, so nothing else exists.
|
|
1209
|
+
loader.setup
|
|
1210
|
+
loader.eager_load
|
|
1211
|
+
%w[
|
|
1212
|
+
Traject::SolrPool::Connection
|
|
1213
|
+
Traject::SolrPool::SolrJsonWriter
|
|
1214
|
+
].each { |c| Object.const_get(c) }
|
|
1215
|
+
print 'ZEITWERK_OK'
|
|
1216
|
+
RUBY
|
|
1217
|
+
end
|
|
1218
|
+
|
|
1219
|
+
it 'eager-loads cleanly under a real Zeitwerk loader in a clean process' do
|
|
1220
|
+
expect(run_in_clean_process(probe)).to include('ZEITWERK_OK')
|
|
1221
|
+
end
|
|
1222
|
+
|
|
1223
|
+
it 'reports no naming errors from Zeitwerk' do
|
|
1224
|
+
expect(run_in_clean_process(probe)).not_to match(/Zeitwerk::|NameError|expected file/)
|
|
1225
|
+
end
|
|
1226
|
+
|
|
1227
|
+
def run_in_clean_process(source)
|
|
1228
|
+
require 'open3'
|
|
1229
|
+
out, status = Open3.capture2e('bundle', 'exec', 'ruby', '-e', source, chdir: gem_root)
|
|
1230
|
+
raise "probe failed (#{status.exitstatus}):\n#{out}" unless status.success?
|
|
1231
|
+
|
|
1232
|
+
out
|
|
1233
|
+
end
|
|
1234
|
+
end
|
|
1235
|
+
```
|
|
1236
|
+
|
|
1237
|
+
- [ ] **Step 3: Run the Zeitwerk spec**
|
|
1238
|
+
|
|
1239
|
+
Run: `bundle exec rspec spec/integration/zeitwerk_compliance_spec.rb`
|
|
1240
|
+
Expected: PASS. If it fails because `traject/solr_pool.rb` requires `traject` (pulling the whole traject tree into the loader's managed dir), add `loader.do_not_eager_load` for non-owned paths or push_dir only conceptually — the pragmatic fix is: since Zeitwerk manages `lib`, and traject is loaded via `require 'traject'` before setup, `require 'traject'` inside our files is fine; the loader only autoloads files under `lib` matching unresolved constants. If traject's own files under a *different* gem dir are not under our `lib`, they are not managed. This should pass as written; if a NameError names a traject constant, add that file to `loader.ignore` and note it in a comment.
|
|
1241
|
+
|
|
1242
|
+
- [ ] **Step 4: Write the Rails-compatibility spec**
|
|
1243
|
+
|
|
1244
|
+
Create `spec/integration/rails_compatibility_spec.rb`:
|
|
1245
|
+
|
|
1246
|
+
```ruby
|
|
1247
|
+
# frozen_string_literal: true
|
|
1248
|
+
|
|
1249
|
+
require 'spec_helper'
|
|
1250
|
+
require 'active_support'
|
|
1251
|
+
require 'active_support/core_ext/object/blank'
|
|
1252
|
+
|
|
1253
|
+
# Confirms the writer coexists with activesupport loaded (as in a Rails app) and
|
|
1254
|
+
# behaves inside a plain service object. No full Rails boot; activesupport only.
|
|
1255
|
+
RSpec.describe 'Rails compatibility', :integration, :solr_stub do
|
|
1256
|
+
it 'loads alongside activesupport without redefining core behaviour' do
|
|
1257
|
+
expect(''.blank?).to be(true)
|
|
1258
|
+
expect(defined?(Traject::SolrPool::SolrJsonWriter)).to eq('constant')
|
|
1259
|
+
end
|
|
1260
|
+
|
|
1261
|
+
it 'indexes from a Rails-style service object' do
|
|
1262
|
+
stub = stub_solr_update('http://solr.test:8983/solr/core/update/json')
|
|
1263
|
+
service = Class.new do
|
|
1264
|
+
def call
|
|
1265
|
+
w = Traject::SolrPool::SolrJsonWriter.new(
|
|
1266
|
+
'solr.url' => 'http://solr.test:8983/solr/core',
|
|
1267
|
+
'solr_writer.thread_pool' => 0, 'solr_writer.batch_size' => 1
|
|
1268
|
+
)
|
|
1269
|
+
w.put(Traject::Indexer::Context.new(output_hash: { 'id' => 'svc-1' }))
|
|
1270
|
+
w.close
|
|
1271
|
+
end
|
|
1272
|
+
end
|
|
1273
|
+
service.new.call
|
|
1274
|
+
expect(stub).to have_been_requested
|
|
1275
|
+
end
|
|
1276
|
+
end
|
|
1277
|
+
```
|
|
1278
|
+
|
|
1279
|
+
- [ ] **Step 5: Run the Rails spec**
|
|
1280
|
+
|
|
1281
|
+
Run: `bundle exec rspec spec/integration/rails_compatibility_spec.rb`
|
|
1282
|
+
Expected: PASS (2 examples).
|
|
1283
|
+
|
|
1284
|
+
- [ ] **Step 6: Run the full suite and RuboCop**
|
|
1285
|
+
|
|
1286
|
+
Run: `bundle exec rspec && bundle exec rubocop`
|
|
1287
|
+
Expected: all green, RuboCop clean.
|
|
1288
|
+
|
|
1289
|
+
- [ ] **Step 7: Commit**
|
|
1290
|
+
|
|
1291
|
+
```bash
|
|
1292
|
+
git add spec/integration/zeitwerk_compliance_spec.rb spec/integration/rails_compatibility_spec.rb .rubocop.yml
|
|
1293
|
+
git commit -m "Add Zeitwerk and Rails compatibility integration specs"
|
|
1294
|
+
```
|
|
1295
|
+
|
|
1296
|
+
---
|
|
1297
|
+
|
|
1298
|
+
## Task 9: Documentation and Rakefile
|
|
1299
|
+
|
|
1300
|
+
**Files:**
|
|
1301
|
+
- Modify: `README.md`
|
|
1302
|
+
- Modify: `Rakefile`
|
|
1303
|
+
- Create: `docs/usage.md`
|
|
1304
|
+
|
|
1305
|
+
**Interfaces:**
|
|
1306
|
+
- Produces: user-facing docs for `writer_class_name` usage, settings, the temporary edge-traject dependency, and a `bundle:audit` CI wiring.
|
|
1307
|
+
|
|
1308
|
+
- [ ] **Step 1: Wire bundler-audit into the default Rake task**
|
|
1309
|
+
|
|
1310
|
+
Replace `Rakefile` contents:
|
|
1311
|
+
|
|
1312
|
+
```ruby
|
|
1313
|
+
# frozen_string_literal: true
|
|
1314
|
+
|
|
1315
|
+
require 'bundler/gem_tasks'
|
|
1316
|
+
require 'rspec/core/rake_task'
|
|
1317
|
+
require 'rubocop/rake_task'
|
|
1318
|
+
require 'bundler/audit/task'
|
|
1319
|
+
|
|
1320
|
+
RSpec::Core::RakeTask.new(:spec)
|
|
1321
|
+
RuboCop::RakeTask.new
|
|
1322
|
+
|
|
1323
|
+
task ci: %i[bundle:audit:check rubocop spec]
|
|
1324
|
+
task default: :ci
|
|
1325
|
+
```
|
|
1326
|
+
|
|
1327
|
+
- [ ] **Step 2: Verify the Rake tasks run**
|
|
1328
|
+
|
|
1329
|
+
Run: `bundle exec rake -T`
|
|
1330
|
+
Expected: lists `spec`, `rubocop`, `bundle:audit:check`, `ci`, `default`. Then run `bundle exec rake ci` — expected green (audit offline, RuboCop clean, RSpec green).
|
|
1331
|
+
|
|
1332
|
+
- [ ] **Step 3: Rewrite README.md**
|
|
1333
|
+
|
|
1334
|
+
Replace the placeholder README with real content. Required sections (H1 `# Traject::SolrPool`, then): what it is (pooled drop-in Solr JSON writer), installation, the temporary edge-traject dependency note and how to remove it, `writer_class_name` usage example, the settings table (copy the settings section from the design spec), dropped `solr_json_writer.*` settings note, thread-safety/Sidekiq/Zeitwerk statement, development (`bundle exec rake ci`), and the never-push policy under contributing. Use single-quoted Ruby in fenced `ruby` blocks. Example usage block to include:
|
|
1335
|
+
|
|
1336
|
+
````markdown
|
|
1337
|
+
```ruby
|
|
1338
|
+
require 'traject/solr_pool'
|
|
1339
|
+
|
|
1340
|
+
settings do
|
|
1341
|
+
provide 'writer_class_name', 'Traject::SolrPool::SolrJsonWriter'
|
|
1342
|
+
provide 'solr.url', 'http://localhost:8983/solr/my_core'
|
|
1343
|
+
provide 'solr_writer.thread_pool', 4
|
|
1344
|
+
provide 'solr_pool.pool_size', 5
|
|
1345
|
+
end
|
|
1346
|
+
```
|
|
1347
|
+
````
|
|
1348
|
+
|
|
1349
|
+
- [ ] **Step 4: Write docs/usage.md**
|
|
1350
|
+
|
|
1351
|
+
Create `docs/usage.md` with the full settings reference (the table from the design spec), the error-handling behaviour (`BadHttpResponse`, default skippable exceptions), and the pool-stays-warm-on-close note. One H1, sentence-case headings, fenced code with language tags.
|
|
1352
|
+
|
|
1353
|
+
- [ ] **Step 5: RuboCop on Rakefile**
|
|
1354
|
+
|
|
1355
|
+
Run: `bundle exec rubocop Rakefile`
|
|
1356
|
+
Expected: clean.
|
|
1357
|
+
|
|
1358
|
+
- [ ] **Step 6: Commit**
|
|
1359
|
+
|
|
1360
|
+
```bash
|
|
1361
|
+
git add README.md Rakefile docs/usage.md
|
|
1362
|
+
git commit -m "Document usage and settings; wire bundler-audit into rake ci"
|
|
1363
|
+
```
|
|
1364
|
+
|
|
1365
|
+
---
|
|
1366
|
+
|
|
1367
|
+
## Self-Review
|
|
1368
|
+
|
|
1369
|
+
**Spec coverage:**
|
|
1370
|
+
- Goal (pooled drop-in writer) → Tasks 2–5. ✓
|
|
1371
|
+
- Reusable Connection seam → Task 2. ✓
|
|
1372
|
+
- Settings vocabulary + `solr_pool.pool_size` → Task 3. ✓
|
|
1373
|
+
- Output path + batch fallback → Task 4. ✓
|
|
1374
|
+
- Error mapping (`BadHttpResponse` + http.rb-native skippable list) → Task 4. ✓
|
|
1375
|
+
- `commit`/`delete`/`delete_all!`, close leaves pool warm → Task 5. ✓
|
|
1376
|
+
- Thread safety → Task 6. ✓
|
|
1377
|
+
- Sidekiq >= 8 → Task 7. ✓
|
|
1378
|
+
- Zeitwerk + Rails → Task 8. ✓
|
|
1379
|
+
- Test-only dep grouping, single-quote RuboCop → Task 1 (config already set in brainstorming commit). ✓
|
|
1380
|
+
- Docs + bundler-audit → Task 9. ✓
|
|
1381
|
+
- Temporary edge-traject dependency documented → Tasks 1 (gemspec NOTE) + 9 (README). ✓
|
|
1382
|
+
|
|
1383
|
+
**Placeholder scan:** No TBD/TODO left as work items; the gemspec/README "TODO" strings are explicitly replaced in Tasks 1 and 9.
|
|
1384
|
+
|
|
1385
|
+
**Type consistency:** `Connection#post(path, body:)` / `#get(path, params:)` defined in Task 2 and consumed identically in Tasks 4–5. `request_path` (Task 3) used in Tasks 4–5. `BadHttpResponse#response` is an http.rb response (`#code`/`#to_s`) consistently. `skipped_record_count`, `MaxSkippedRecordsExceeded` names consistent across Tasks 4–5.
|
|
1386
|
+
|
|
1387
|
+
**Known verification points flagged inline (not placeholders):** `Registry.close_all` vs `reset!` (Task 1 Step 7), http.rb `params: {}` on GET (Task 2 Step 5), Sidekiq 8 inline load without Redis (Task 7 Step 4), Zeitwerk ignore set (Task 8 Step 3). Each has a concrete fallback.
|