traject-solr_pool 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.github/workflows/ci.yml +34 -0
- data/CLAUDE.md +237 -0
- data/LICENSE.txt +21 -0
- data/README.md +167 -0
- data/Rakefile +21 -0
- data/docs/agents/README.md +41 -0
- data/docs/skills/README.md +42 -0
- data/docs/superpowers/plans/2026-07-01-traject-solr-pool-writer.md +1387 -0
- data/docs/superpowers/specs/2026-07-01-traject-solr-pool-writer-design.md +265 -0
- data/docs/usage.md +122 -0
- data/lib/traject/solr_pool/connection.rb +80 -0
- data/lib/traject/solr_pool/solr_json_writer.rb +303 -0
- data/lib/traject/solr_pool/version.rb +7 -0
- data/lib/traject/solr_pool.rb +15 -0
- data/sig/traject/solr_pool.rbs +6 -0
- metadata +95 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 5ba919dd1768bd1f05ff1e29af8236790aee47bd0333f97c9be2cd66b691fa0a
|
|
4
|
+
data.tar.gz: 01e411c06eb9dd7f101630e42829c0dddacb13931ee1bb0c38f4966383fdcb5a
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 131acdc7092f6f935ba9d16b1aa4c50047b86694d06dc4ba571c4d8415624ff461efe549351e4c8db76d106ab58539f469fd3491bc8bd709d1875d65eaa73794
|
|
7
|
+
data.tar.gz: '0687c890c8944c6b2b024c49b09748f78c8626a7a8531ce0aac553cc5c5bd173537e7ef9cf0a30f23c33976022e7a41f40d2587cb4056bfe5fb23a1e0f8cd040'
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
name: CI
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main]
|
|
6
|
+
pull_request:
|
|
7
|
+
|
|
8
|
+
jobs:
|
|
9
|
+
test:
|
|
10
|
+
runs-on: ubuntu-latest
|
|
11
|
+
strategy:
|
|
12
|
+
fail-fast: false
|
|
13
|
+
matrix:
|
|
14
|
+
ruby: ['3.3', '3.4']
|
|
15
|
+
steps:
|
|
16
|
+
- uses: actions/checkout@v4
|
|
17
|
+
- uses: ruby/setup-ruby@v1
|
|
18
|
+
with:
|
|
19
|
+
ruby-version: ${{ matrix.ruby }}
|
|
20
|
+
bundler-cache: true
|
|
21
|
+
# `rake ci` runs bundle:audit:check OFFLINE. No advisory DB is cloned in
|
|
22
|
+
# this job, so that step no-ops; the authoritative CVE scan is the
|
|
23
|
+
# `security` job below, which refreshes the DB over the network.
|
|
24
|
+
- run: bundle exec rake ci
|
|
25
|
+
|
|
26
|
+
security:
|
|
27
|
+
runs-on: ubuntu-latest
|
|
28
|
+
steps:
|
|
29
|
+
- uses: actions/checkout@v4
|
|
30
|
+
- uses: ruby/setup-ruby@v1
|
|
31
|
+
with:
|
|
32
|
+
ruby-version: '3.4'
|
|
33
|
+
bundler-cache: true
|
|
34
|
+
- run: bundle exec rake audit
|
data/CLAUDE.md
ADDED
|
@@ -0,0 +1,237 @@
|
|
|
1
|
+
# CLAUDE.md — traject-solr_pool
|
|
2
|
+
|
|
3
|
+
## Project Goal
|
|
4
|
+
|
|
5
|
+
This gem is a **plugin for [traject](https://github.com/traject/traject)** that
|
|
6
|
+
lets traject writers (and future readers) send Solr/HTTP traffic over a
|
|
7
|
+
**persistent, pooled, thread-safe, Fiber-scheduler-aware HTTP client** instead
|
|
8
|
+
of opening a fresh connection per request.
|
|
9
|
+
|
|
10
|
+
The pooling itself is **not** reimplemented here. It is provided by the
|
|
11
|
+
`http_connection_pool` gem (a sibling project), which keeps one pool of
|
|
12
|
+
persistent `HTTP::Session` connections per origin, credential-isolated and safe
|
|
13
|
+
under heavy concurrency. This gem is the **traject-facing adapter**: it exposes
|
|
14
|
+
a writer (initially a drop-in replacement for `Traject::SolrJsonWriter`) that
|
|
15
|
+
speaks traject's writer contract while borrowing connections from that pool.
|
|
16
|
+
|
|
17
|
+
The deliverable is a Solr JSON writer that:
|
|
18
|
+
|
|
19
|
+
- registers via traject's `writer_class_name` setting,
|
|
20
|
+
- honours the existing `solr.*` / `solr_writer.*` settings surface so it is a
|
|
21
|
+
drop-in for the stock writer wherever practical,
|
|
22
|
+
- and routes every HTTP call through an `http_connection_pool` pool keyed by
|
|
23
|
+
the Solr origin.
|
|
24
|
+
|
|
25
|
+
A reader is explicitly **future scope** — design the code so a reader can reuse
|
|
26
|
+
the same pooling seam later, but do not build it until asked.
|
|
27
|
+
|
|
28
|
+
---
|
|
29
|
+
|
|
30
|
+
## Dependency Situation (resolved)
|
|
31
|
+
|
|
32
|
+
`http_connection_pool` requires `http ~> 6.0`. Earlier released traject gems
|
|
33
|
+
capped their dependency at `http < 6`, so the two could not co-install from
|
|
34
|
+
RubyGems, and the project bridged the gap with a Gemfile `path:` override to a
|
|
35
|
+
local edge checkout.
|
|
36
|
+
|
|
37
|
+
- **traject 3.9.0 (released 2026-07-21) relaxed the cap** to `http >= 3.0,
|
|
38
|
+
< 7`, so traject now co-installs with `http_connection_pool` straight from
|
|
39
|
+
RubyGems. The gemspec pins `traject '~> 3.9'`.
|
|
40
|
+
- The `path:` override is **no longer needed**. It is commented out in the
|
|
41
|
+
`Gemfile` (kept as a hook in case a local traject checkout is ever needed
|
|
42
|
+
again, e.g. to test an unreleased fix) — do not re-enable it for normal work.
|
|
43
|
+
- Do **not** hardcode absolute local filesystem paths anywhere in committed
|
|
44
|
+
source, specs, or docs. Any local override lives in the `Gemfile` and refers
|
|
45
|
+
to a sibling checkout by relative path.
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## Non-Negotiable Constraints
|
|
50
|
+
|
|
51
|
+
### 1. Git and publishing
|
|
52
|
+
- **You may `git commit`. You may NEVER `git push`** — recommend the push
|
|
53
|
+
command and let the user run it. This also covers `gem push`, `gem release`,
|
|
54
|
+
`rake release`, and `git push --force` (all forbidden; user-only).
|
|
55
|
+
- Follow `~/.gitignore` hygiene: never stage `Gemfile.lock` (it is gitignored
|
|
56
|
+
here), never use `git add -A`/`git add .` — add files by name, and check
|
|
57
|
+
`.gitignore` before staging.
|
|
58
|
+
- Verify no secrets (Solr credentials, tokens) are ever committed.
|
|
59
|
+
|
|
60
|
+
### 2. Test-driven development
|
|
61
|
+
- **RSpec, TDD.** Write a failing spec before implementation code. See the
|
|
62
|
+
RSpec section below.
|
|
63
|
+
- The whole point of the gem is concurrency-safe pooling, so writer behaviour
|
|
64
|
+
under a multi-thread `solr_writer.thread_pool` must be covered by specs, not
|
|
65
|
+
assumed.
|
|
66
|
+
|
|
67
|
+
### 3. Thread safety, Sidekiq, Zeitwerk, and Rails compatibility (MUST)
|
|
68
|
+
This gem must be as safe and portable to embed as `http_connection_pool` is.
|
|
69
|
+
These are hard requirements, not aspirations:
|
|
70
|
+
|
|
71
|
+
- **Thread safety is mandatory.** The writer is exercised concurrently — both
|
|
72
|
+
by traject's own `solr_writer.thread_pool` and by hosting apps. All shared
|
|
73
|
+
state must be safe under concurrent access. Prefer `concurrent-ruby`
|
|
74
|
+
primitives (`Concurrent::AtomicFixnum`, `Concurrent::Map`, etc.) as the stock
|
|
75
|
+
`SolrJsonWriter` does; never guard the hot path with a coarse global `Mutex`.
|
|
76
|
+
Let the underlying `http_connection_pool` own connection checkout/return —
|
|
77
|
+
never hold a borrowed connection across threads.
|
|
78
|
+
- **Sidekiq >= 8 compatibility is a target.** The writer must be usable from
|
|
79
|
+
inside a Sidekiq >= 8 worker: instances may be created and torn down per job,
|
|
80
|
+
many workers/threads run concurrently, and the process forks. Rely on
|
|
81
|
+
`http_connection_pool`'s fork-safety (`connection_pool >= 2.5`
|
|
82
|
+
`auto_reload_after_fork`); do not cache raw sockets or connection objects
|
|
83
|
+
across a fork in this gem. Cover the "used from a background job / forked
|
|
84
|
+
worker" path with specs, mirroring the sibling gem's background-job specs.
|
|
85
|
+
- **Zeitwerk + Rails compatibility is a target.** The gem must coexist cleanly
|
|
86
|
+
with a Rails app's Zeitwerk loader and not corrupt eager-loading. Keep the
|
|
87
|
+
file/constant layout Zeitwerk-conformant (file path matches constant name)
|
|
88
|
+
even though traject itself loads via plain `require`. No Rails **runtime**
|
|
89
|
+
dependency — Rails/Zeitwerk are **test-only** concerns, verified by
|
|
90
|
+
integration specs (mirror `http_connection_pool`'s Rails-compat and
|
|
91
|
+
clean-subprocess Zeitwerk specs). Do not add `rails`, `activesupport`, or
|
|
92
|
+
`zeitwerk` to `spec.add_dependency`.
|
|
93
|
+
|
|
94
|
+
### 4. Respect the traject writer contract
|
|
95
|
+
A traject writer is any class that responds to:
|
|
96
|
+
- `initialize(settings)` — receives a `Traject::Indexer::Settings` (a Hash).
|
|
97
|
+
- `put(context)` — enqueue one `Traject::Indexer::Context` for output.
|
|
98
|
+
- `close` — flush remaining work, shut down threads, optionally commit.
|
|
99
|
+
|
|
100
|
+
`SolrJsonWriter` additionally provides `flush`, `commit`, `delete`,
|
|
101
|
+
`delete_all!`, and `skipped_record_count`. Mirror this surface so the writer is
|
|
102
|
+
a genuine drop-in. **Do not invent a new settings vocabulary** — reuse the
|
|
103
|
+
documented `solr.*` and `solr_writer.*` keys. New settings, if unavoidable, go
|
|
104
|
+
under a clearly namespaced prefix and must be documented.
|
|
105
|
+
|
|
106
|
+
### 5. Use the pool through its public API
|
|
107
|
+
- Integrate via `http_connection_pool`'s public surface (the `Connectable`
|
|
108
|
+
mixin / `Registry` / `Pool`). Do not reach into its internals or reimplement
|
|
109
|
+
keep-alive, mutexes, or socket handling.
|
|
110
|
+
- `base_url` for a pool is an **origin** (`scheme://host:port`); request paths
|
|
111
|
+
(e.g. `/solr/<core>/update/json`) are **relative** and passed per request.
|
|
112
|
+
- Solr basic-auth credentials and content-type headers belong in
|
|
113
|
+
`pool_options` (e.g. `headers:`), never baked into the origin. Pools are
|
|
114
|
+
keyed by a SHA-256 digest of `(origin, options)`, so distinct credentials get
|
|
115
|
+
distinct pools automatically — never defeat that isolation.
|
|
116
|
+
- `pool_options` **replaces, does not merge**. If merge semantics are needed,
|
|
117
|
+
the caller composes the hash explicitly.
|
|
118
|
+
- `http_connection_pool` rejects non-scalar option values such as
|
|
119
|
+
`ssl_context:` (raises `OptionKeyError`). Do not pass one; translate SSL
|
|
120
|
+
needs through supported options or document the limitation.
|
|
121
|
+
|
|
122
|
+
### 6. No leakage of credentials in logs or errors
|
|
123
|
+
- The stock writer deliberately strips embedded user/password from the update
|
|
124
|
+
URL before logging (`determine_solr_update_url`). Preserve that: never log or
|
|
125
|
+
raise a message containing basic-auth credentials or auth headers.
|
|
126
|
+
|
|
127
|
+
### 7. Ruby and portability
|
|
128
|
+
- `required_ruby_version >= 3.3.0`. Target and test on **MRI (CRuby)**. JRuby is
|
|
129
|
+
not verified — do not claim JRuby support until it is exercised.
|
|
130
|
+
|
|
131
|
+
---
|
|
132
|
+
|
|
133
|
+
## Code Style
|
|
134
|
+
|
|
135
|
+
- **RuboCop is authoritative and must be clean before any commit.** Config is
|
|
136
|
+
`.rubocop.yml`; it enforces **single-quoted** non-interpolated string literals
|
|
137
|
+
(`Style/StringLiterals: single_quotes`) — follow it. Use double quotes only
|
|
138
|
+
when the string interpolates.
|
|
139
|
+
- Every Ruby file begins with `# frozen_string_literal: true`.
|
|
140
|
+
- Run `bundle exec rubocop -a` to auto-fix correctable offences. Never silence a
|
|
141
|
+
cop with an inline `# rubocop:disable` unless there is genuinely no
|
|
142
|
+
alternative — restructure or extract a method instead.
|
|
143
|
+
- Comments explain *why* (hidden constraints, invariants, workarounds), not
|
|
144
|
+
*what*. Prefer `attr_reader` over hand-written getters. Keep methods short.
|
|
145
|
+
- Plugins are declared with `plugins:` syntax in `.rubocop.yml`:
|
|
146
|
+
`rubocop-performance`, `rubocop-rake`, `rubocop-rspec`.
|
|
147
|
+
|
|
148
|
+
---
|
|
149
|
+
|
|
150
|
+
## RSpec
|
|
151
|
+
|
|
152
|
+
- **Never use apostrophes in `it`/`describe`/`context` strings** — `it 'a b's c'`
|
|
153
|
+
is a Ruby `SyntaxError`. Rephrase to avoid the apostrophe.
|
|
154
|
+
- Helper methods/classes for specs go in a `spec/support` module included by tag
|
|
155
|
+
— never define a top-level `class`/`def` in a spec file (it leaks a global
|
|
156
|
+
constant). `spec/support/**/*.rb` should be auto-required by `spec_helper.rb`.
|
|
157
|
+
- Use `after do` (not `after(:each) do`) for teardown. Reset any global pool
|
|
158
|
+
registry between examples so state never leaks (mirror the sibling gem's
|
|
159
|
+
`Registry.reset!` teardown pattern).
|
|
160
|
+
- Stub Solr HTTP at the pool/connection boundary so unit specs do not require a
|
|
161
|
+
live Solr. Reserve live-Solr exercises for clearly tagged integration specs.
|
|
162
|
+
- Concurrency is a first-class behaviour: cover `put`/`send_batch`/`close` under
|
|
163
|
+
a threaded `solr_writer.thread_pool`, not just the single-thread path.
|
|
164
|
+
- Rails/Zeitwerk/Sidekiq compatibility lives in tagged integration specs (see
|
|
165
|
+
constraint 3); keep them out of the fast unit path.
|
|
166
|
+
|
|
167
|
+
---
|
|
168
|
+
|
|
169
|
+
## Documentation
|
|
170
|
+
|
|
171
|
+
- **Whenever a change affects behaviour, settings, dependencies, or usage,
|
|
172
|
+
update `README.md` and any relevant files under `docs/` in the same change.**
|
|
173
|
+
This is standing — do not wait to be asked.
|
|
174
|
+
- Document the settings the writer honours, the temporary edge-traject
|
|
175
|
+
dependency and how to remove it, and a `writer_class_name` usage example.
|
|
176
|
+
- Markdown: one `#` H1 per file, sentence-case headings in nesting order, fenced
|
|
177
|
+
code blocks with a language tag, blank lines around headings/lists/fences.
|
|
178
|
+
|
|
179
|
+
---
|
|
180
|
+
|
|
181
|
+
## File / Constant Layout
|
|
182
|
+
|
|
183
|
+
Namespaced under `Traject::SolrPool` (traject convention; traject does not use
|
|
184
|
+
Zeitwerk itself — load with `require`/`require_relative`, not autoloading). The
|
|
185
|
+
layout must still be **Zeitwerk-conformant** (file path matches constant name)
|
|
186
|
+
so a host Rails app's loader stays happy — verify with an integration spec.
|
|
187
|
+
|
|
188
|
+
```
|
|
189
|
+
lib/
|
|
190
|
+
traject/solr_pool.rb # entry point
|
|
191
|
+
traject/solr_pool/
|
|
192
|
+
version.rb # Traject::SolrPool::VERSION
|
|
193
|
+
... # writer + supporting classes (TBD in design)
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
`version.rb` defines `Traject::SolrPool::VERSION`. Keep the constant tree
|
|
197
|
+
matching the file tree.
|
|
198
|
+
|
|
199
|
+
---
|
|
200
|
+
|
|
201
|
+
## Rake Tasks
|
|
202
|
+
|
|
203
|
+
| Task | What it does |
|
|
204
|
+
| ------------------------- | ----------------------------------------------------- |
|
|
205
|
+
| `rake` / `rake ci` | `bundle:audit:check` (offline) → RuboCop → RSpec |
|
|
206
|
+
| `rake audit` | `bundle:audit:update` (network) → `bundle:audit:check`|
|
|
207
|
+
| `rake spec` | RSpec only |
|
|
208
|
+
| `rake rubocop` | RuboCop only |
|
|
209
|
+
| `rake bundle:audit:check` | Offline CVE scan |
|
|
210
|
+
|
|
211
|
+
`rake ci` must run clean before any commit. Never bypass bundler-audit.
|
|
212
|
+
|
|
213
|
+
## Continuous integration
|
|
214
|
+
|
|
215
|
+
`.github/workflows/ci.yml` runs on push to `main` and on PRs: a `test` matrix
|
|
216
|
+
over MRI Ruby `3.3` and `3.4` running `rake ci` (offline audit + RuboCop +
|
|
217
|
+
RSpec), plus a separate `security` job running `rake audit` (network advisory-DB
|
|
218
|
+
refresh + check). MRI only — mirrors `http_connection_pool`'s CI. `Gemfile.lock`
|
|
219
|
+
is gitignored, so CI resolves dependencies fresh each run.
|
|
220
|
+
|
|
221
|
+
Automated publishing (`release.yml` / OIDC Trusted Publishing) is planned but
|
|
222
|
+
not yet wired; the initial release is published manually by the maintainer.
|
|
223
|
+
|
|
224
|
+
---
|
|
225
|
+
|
|
226
|
+
## Key Decisions (do not reverse without discussion)
|
|
227
|
+
|
|
228
|
+
1. **Pooling is delegated to `http_connection_pool`** — this gem is a thin
|
|
229
|
+
traject adapter, not a second pooling implementation.
|
|
230
|
+
2. **Drop-in for `SolrJsonWriter`** — reuse its settings vocabulary and public
|
|
231
|
+
method surface; do not fork a new configuration language.
|
|
232
|
+
3. **Thread-safe, Sidekiq >= 8, Zeitwerk, and Rails compatible** — first-class
|
|
233
|
+
requirements verified by specs, matching `http_connection_pool`'s guarantees.
|
|
234
|
+
4. **Released traject only** — pin `traject '~> 3.9'` (3.9.0 lifted the
|
|
235
|
+
`http < 6` cap). The old edge `path:` override is commented out in the
|
|
236
|
+
Gemfile; do not re-enable it except to test an unreleased traject fix.
|
|
237
|
+
5. **Reader is future scope** — leave a clean seam, build only when asked.
|
data/LICENSE.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
The MIT License (MIT)
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 bbarberBPL
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
|
13
|
+
all copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,167 @@
|
|
|
1
|
+
# Traject::SolrPool
|
|
2
|
+
|
|
3
|
+
`Traject::SolrPool::SolrJsonWriter` is a drop-in pooled replacement for the
|
|
4
|
+
stock `Traject::SolrJsonWriter`. It routes every Solr update through a
|
|
5
|
+
persistent, credential-isolated, thread-safe HTTP connection pool provided by
|
|
6
|
+
[`http_connection_pool`][hcp] instead of opening a fresh `HTTPClient`
|
|
7
|
+
connection per request.
|
|
8
|
+
|
|
9
|
+
It reuses the stock writer's `solr.*` / `solr_writer.*` settings vocabulary, so
|
|
10
|
+
it is a drop-in wherever those settings are practical. The writer is:
|
|
11
|
+
|
|
12
|
+
- thread-safe (shared state uses `concurrent-ruby` primitives; a borrowed
|
|
13
|
+
connection is never held across threads),
|
|
14
|
+
- usable from Sidekiq >= 8 workers (fork-safe via `connection_pool >= 2.5`),
|
|
15
|
+
- Zeitwerk-conformant, and
|
|
16
|
+
- compatible with Rails (no Rails runtime dependency).
|
|
17
|
+
|
|
18
|
+
All four properties are verified by the test suite (unit, concurrency, Sidekiq,
|
|
19
|
+
Zeitwerk, and Rails-coexistence specs).
|
|
20
|
+
|
|
21
|
+
[hcp]: https://github.com/bbarberBPL/http_connection_pool
|
|
22
|
+
|
|
23
|
+
## Installation
|
|
24
|
+
|
|
25
|
+
Add the gem to your application's Gemfile:
|
|
26
|
+
|
|
27
|
+
```ruby
|
|
28
|
+
gem 'traject-solr_pool'
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
Then install:
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
bundle install
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
## Requirements
|
|
38
|
+
|
|
39
|
+
- Ruby >= 3.3
|
|
40
|
+
- `traject` >= 3.9 (3.9.0 relaxed its HTTP cap to `http >= 3.0, < 7`, so it
|
|
41
|
+
co-installs with `http_connection_pool`'s `http ~> 6.0` directly from
|
|
42
|
+
RubyGems — no edge checkout needed)
|
|
43
|
+
- `http_connection_pool` >= 0.1.1
|
|
44
|
+
|
|
45
|
+
Earlier releases of `traject` capped their HTTP dependency at `http < 6`, which
|
|
46
|
+
collided with `http_connection_pool`'s `http ~> 6.0`; the Gemfile once carried a
|
|
47
|
+
`path:` override to a local edge checkout to work around that. It is no longer
|
|
48
|
+
required and the override is commented out.
|
|
49
|
+
|
|
50
|
+
## Usage
|
|
51
|
+
|
|
52
|
+
Register the writer with traject's `writer_class_name` setting:
|
|
53
|
+
|
|
54
|
+
```ruby
|
|
55
|
+
require 'traject/solr_pool'
|
|
56
|
+
|
|
57
|
+
settings do
|
|
58
|
+
provide 'writer_class_name', 'Traject::SolrPool::SolrJsonWriter'
|
|
59
|
+
provide 'solr.url', 'http://localhost:8983/solr/my_core'
|
|
60
|
+
provide 'solr_writer.thread_pool', 4
|
|
61
|
+
provide 'solr_pool.pool_size', 5
|
|
62
|
+
end
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
The origin (`scheme://host:port`) derived from `solr.url` becomes the pool's
|
|
66
|
+
`base_url`; the remainder of the URL becomes the relative request path. Basic
|
|
67
|
+
auth credentials are sent as an `Authorization` header, never baked into the
|
|
68
|
+
origin, so distinct credentials resolve to distinct pools and never share
|
|
69
|
+
connections.
|
|
70
|
+
|
|
71
|
+
## Settings
|
|
72
|
+
|
|
73
|
+
The writer honours the stock `solr.*` / `solr_writer.*` vocabulary, plus one
|
|
74
|
+
new namespaced setting.
|
|
75
|
+
|
|
76
|
+
| Setting | Default | Role |
|
|
77
|
+
| --- | --- | --- |
|
|
78
|
+
| `solr.url` | (required unless `solr.update_url` set) | Base Solr URL; `/update/json` is derived from it |
|
|
79
|
+
| `solr.update_url` | derived from `solr.url` | Full update handler URL; used verbatim when provided |
|
|
80
|
+
| `solr_writer.batch_size` | `100` | Documents per batched update |
|
|
81
|
+
| `solr_writer.thread_pool` | `1` | Number of background writer threads |
|
|
82
|
+
| `solr_writer.max_skipped` | `0` | Skip tolerance before `MaxSkippedRecordsExceeded` (negative disables the cap) |
|
|
83
|
+
| `solr_writer.skippable_exceptions` | http.rb-native list (see below) | Exceptions treated as skippable during per-record retry |
|
|
84
|
+
| `solr_writer.commit_on_close` | `false` | Send a commit when the writer closes (legacy `solrj_writer.commit_on_close` also honoured) |
|
|
85
|
+
| `solr_writer.solr_update_args` | none | Query params (e.g. `{ 'commitWithin' => 1000 }`) applied to every update and delete request |
|
|
86
|
+
| `solr_writer.commit_solr_update_args` | `{ 'commit' => 'true' }` | Query params for the commit request |
|
|
87
|
+
| `solr_writer.commit_timeout` | `600` (10 min) | Read timeout applied to the commit request only, so a slow commit is not cut off by a short `http_timeout` |
|
|
88
|
+
| `solr_writer.basic_auth_user` | embedded URI user | Basic-auth user (overrides credentials embedded in the URL) |
|
|
89
|
+
| `solr_writer.basic_auth_password` | embedded URI password | Basic-auth password (overrides credentials embedded in the URL) |
|
|
90
|
+
| `solr_writer.http_timeout` | none | Per-request HTTP timeout, passed to the pool connection |
|
|
91
|
+
| `solr_writer.pool_timeout` | pool default | Max time to wait for a connection checkout from the pool |
|
|
92
|
+
| `solr_pool.pool_size` | `solr_writer.thread_pool` + 1 | Pool capacity, sized so no writer thread starves on checkout while the caller thread can still flush/commit |
|
|
93
|
+
|
|
94
|
+
### Dropped settings
|
|
95
|
+
|
|
96
|
+
Two `solr_json_writer.*` settings from the stock writer are HTTPClient-specific
|
|
97
|
+
and are not supported:
|
|
98
|
+
|
|
99
|
+
- `solr_json_writer.http_client` — the pool owns connection construction, so
|
|
100
|
+
there is no `HTTPClient` instance to inject.
|
|
101
|
+
- `solr_json_writer.use_packaged_certs` — an `HTTPClient` ssl_config quirk;
|
|
102
|
+
http.rb uses the operating system's certificate store instead.
|
|
103
|
+
|
|
104
|
+
## Error handling
|
|
105
|
+
|
|
106
|
+
- `Traject::SolrPool::SolrJsonWriter::BadHttpResponse` (a `RuntimeError`) is
|
|
107
|
+
raised on any non-200 response. Its `#response` accessor returns the raw
|
|
108
|
+
http.rb response (use `#code` / `#to_s`), and it extracts Solr's JSON
|
|
109
|
+
`error.msg` into the message when present.
|
|
110
|
+
- The default `solr_writer.skippable_exceptions` list is http.rb-native:
|
|
111
|
+
|
|
112
|
+
```ruby
|
|
113
|
+
[
|
|
114
|
+
HTTP::TimeoutError,
|
|
115
|
+
HttpConnectionPool::TimeoutError,
|
|
116
|
+
HTTP::ConnectionError,
|
|
117
|
+
SocketError,
|
|
118
|
+
Errno::ECONNREFUSED,
|
|
119
|
+
Traject::SolrPool::SolrJsonWriter::BadHttpResponse
|
|
120
|
+
]
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
A batch that fails is retried record by record; a record that raises a
|
|
124
|
+
skippable exception is logged and counted, aborting the run only once
|
|
125
|
+
`solr_writer.max_skipped` is exceeded.
|
|
126
|
+
- Setting `solr_writer.skippable_exceptions` explicitly replaces the default
|
|
127
|
+
list entirely with your own.
|
|
128
|
+
- `commit`, `delete`, and `delete_all!` raise raw on failure; they are not part
|
|
129
|
+
of the skip path.
|
|
130
|
+
- Credentials never appear in logs or error messages: auth lives in a header
|
|
131
|
+
and logs show the origin only.
|
|
132
|
+
|
|
133
|
+
## Pool lifecycle
|
|
134
|
+
|
|
135
|
+
`close` flushes any queued records, waits for the background threads, and
|
|
136
|
+
commits when `solr_writer.commit_on_close` is set. It then **leaves the pool
|
|
137
|
+
warm** in the `http_connection_pool` registry so later writers (or a future
|
|
138
|
+
reader) on the same origin reuse it.
|
|
139
|
+
|
|
140
|
+
Teardown is the host application's concern. Close the pools explicitly on
|
|
141
|
+
process shutdown or after a fork, for example:
|
|
142
|
+
|
|
143
|
+
```ruby
|
|
144
|
+
HttpConnectionPool::Registry.instance.close_all
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
## Development
|
|
148
|
+
|
|
149
|
+
Run the full CI pipeline (offline bundler-audit, RuboCop, RSpec):
|
|
150
|
+
|
|
151
|
+
```bash
|
|
152
|
+
bundle exec rake ci
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Individual tasks (`rake spec`, `rake rubocop`, `rake bundle:audit:check`) are
|
|
156
|
+
also available.
|
|
157
|
+
|
|
158
|
+
## Contributing
|
|
159
|
+
|
|
160
|
+
Bug reports and pull requests are welcome. Releasing and pushing are maintainer
|
|
161
|
+
responsibilities — contributors should not push tags or publish gems.
|
|
162
|
+
|
|
163
|
+
## License
|
|
164
|
+
|
|
165
|
+
The gem is available as open source under the terms of the [MIT License][mit].
|
|
166
|
+
|
|
167
|
+
[mit]: https://opensource.org/licenses/MIT
|
data/Rakefile
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'bundler/gem_tasks'
|
|
4
|
+
require 'rspec/core/rake_task'
|
|
5
|
+
require 'rubocop/rake_task'
|
|
6
|
+
require 'bundler/audit/task'
|
|
7
|
+
|
|
8
|
+
RSpec::Core::RakeTask.new(:spec)
|
|
9
|
+
RuboCop::RakeTask.new
|
|
10
|
+
Bundler::Audit::Task.new
|
|
11
|
+
|
|
12
|
+
# The default task runs offline, so `ci` only *checks* against whatever
|
|
13
|
+
# advisory DB is present. `audit` refreshes the DB from the network first;
|
|
14
|
+
# `bundle:audit:check` no-ops gracefully if the DB was never cloned.
|
|
15
|
+
desc 'Refresh the advisory DB, then scan dependencies for known CVEs'
|
|
16
|
+
task audit: ['bundle:audit:update', 'bundle:audit:check']
|
|
17
|
+
|
|
18
|
+
desc 'Audit dependencies (offline), run RuboCop, then the RSpec suite'
|
|
19
|
+
task ci: %i[bundle:audit:check rubocop spec]
|
|
20
|
+
|
|
21
|
+
task default: :ci
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
# Agents
|
|
2
|
+
|
|
3
|
+
Project-specific subagent definitions for `traject-solr_pool`.
|
|
4
|
+
|
|
5
|
+
When a task in this project benefits from a dedicated subagent (a focused
|
|
6
|
+
reviewer, a migration helper, a release auditor, etc.), define it here as one
|
|
7
|
+
Markdown file per agent. Keep each agent's responsibility narrow and its
|
|
8
|
+
instructions self-contained, so it can run without the parent session's
|
|
9
|
+
context.
|
|
10
|
+
|
|
11
|
+
## File convention
|
|
12
|
+
|
|
13
|
+
- One file per agent: `docs/agents/<agent-name>.md`.
|
|
14
|
+
- Start with a short purpose line: what the agent does and when to reach for it.
|
|
15
|
+
- List the tools it needs and the inputs it expects (file paths, not pasted
|
|
16
|
+
context).
|
|
17
|
+
- Describe the report format it must return.
|
|
18
|
+
|
|
19
|
+
## Reusable from the sibling `http_connection_pool` gem
|
|
20
|
+
|
|
21
|
+
This gem shares that gem's thread-safety and security constraints, so several of
|
|
22
|
+
its agents transfer directly (see `../../../http_connection_pool/docs/agents/`).
|
|
23
|
+
Adapt the file paths/tags to this project when reused:
|
|
24
|
+
|
|
25
|
+
- `concurrency-spec-reviewer` — reviews concurrency specs for assertions a race
|
|
26
|
+
can invalidate, then stress-loops them for flakiness. Directly applicable to
|
|
27
|
+
our `:thread_safety` and `:background_jobs` integration specs.
|
|
28
|
+
- `dependency-security-auditor` — audits the dependency tree against primary
|
|
29
|
+
sources (RubyGems API, GitHub advisory DB). Applicable to our gemspec/Gemfile
|
|
30
|
+
whenever a dependency floor changes.
|
|
31
|
+
- `memory-leak-auditor` — hunts unbounded retention with `ObjectSpace`/`GC`
|
|
32
|
+
probes. Applicable to verifying the writer/pool do not accumulate objects
|
|
33
|
+
per-batch or per-job.
|
|
34
|
+
|
|
35
|
+
## Defined agents
|
|
36
|
+
|
|
37
|
+
_None yet. Candidate for this project:_
|
|
38
|
+
|
|
39
|
+
- **writer-parity-reviewer** — diff our `Traject::SolrPool::SolrJsonWriter`
|
|
40
|
+
against the stock `Traject::SolrJsonWriter` to confirm settings/method-surface
|
|
41
|
+
parity and flag any silently-dropped behaviour. Advisory only.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Skills
|
|
2
|
+
|
|
3
|
+
Project-specific skills for `traject-solr_pool`.
|
|
4
|
+
|
|
5
|
+
When a repeatable workflow emerges in this project (a release checklist, a
|
|
6
|
+
dependency-audit routine, a Solr round-trip verification), capture it here as a
|
|
7
|
+
skill so future sessions follow the same steps instead of rediscovering them.
|
|
8
|
+
|
|
9
|
+
## File convention
|
|
10
|
+
|
|
11
|
+
- One directory per skill: `docs/skills/<skill-name>/SKILL.md`.
|
|
12
|
+
- The `SKILL.md` opens with YAML frontmatter (`name`, `description`) describing
|
|
13
|
+
when the skill applies, followed by the steps.
|
|
14
|
+
- Keep steps concrete and ordered; prefer checklists the session can turn into
|
|
15
|
+
todos.
|
|
16
|
+
- Reference supporting files by relative path within the skill directory.
|
|
17
|
+
|
|
18
|
+
## Reusable from the sibling `http_connection_pool` gem
|
|
19
|
+
|
|
20
|
+
See `../../../http_connection_pool/docs/skills/`. These transfer with minor path
|
|
21
|
+
changes:
|
|
22
|
+
|
|
23
|
+
- `dependency-audit` — security sweep that verifies every advisory/version claim
|
|
24
|
+
against primary sources before recommending a change. Reuse verbatim whenever
|
|
25
|
+
a dependency floor changes.
|
|
26
|
+
- `memory-leak-audit` — drives churn and measures retention with
|
|
27
|
+
`ObjectSpace`/`GC` rather than reading code. Adapt the churn driver to enqueue
|
|
28
|
+
writer batches / background jobs against a WebMock-stubbed Solr.
|
|
29
|
+
|
|
30
|
+
## Defined skills
|
|
31
|
+
|
|
32
|
+
_None yet. Candidates for this project:_
|
|
33
|
+
|
|
34
|
+
- **edge-traject-cutover** — _done (2026-07-27)._ traject 3.9.0 lifted the
|
|
35
|
+
`http < 6` cap; the gemspec now pins `traject '~> 3.9'` and the Gemfile
|
|
36
|
+
`path:` override is commented out. Retained here as a record of the cutover
|
|
37
|
+
procedure (verify the published version via the RubyGems API, update the
|
|
38
|
+
gemspec constraint, drop the override, re-run `rake ci`) in case a similar
|
|
39
|
+
edge-dependency bridge is needed again.
|
|
40
|
+
- **solr-roundtrip-verify** — bring up a local Solr, run a real index + commit +
|
|
41
|
+
delete round-trip through the pooled writer, and confirm connection reuse via
|
|
42
|
+
registry stats (mirrors the sibling gem's `examples/solr_update_demo.rb`).
|