active_pipeline 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 07b59b4a629962f09f8a6fde03fece7b3a4e1bcec69e3bf711657e83b1115360
4
+ data.tar.gz: fb6630f5e422714efae2314896925c2becb8f849f774446520ad7e8ff9ac83bd
5
+ SHA512:
6
+ metadata.gz: 7a48549372312bfaa19cb0f21a7960ad93d9cb583db50a675263698683c336f1cfe0fcd70b7497ec0e16126ec7822bd39ab126fb14a10b37c3d0cf2d8350100b
7
+ data.tar.gz: d58eb657403ca873f1721ecb9700318abb2fea358a35be516f1980b3542ba905caa8d0009c4dd065c51080be5ea8d09df8b1c1b87fd9cde4ea276af15b586702
data/CHANGELOG.md ADDED
@@ -0,0 +1,31 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project will be documented in this file.
4
+
5
+ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6
+ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
+
8
+ ## [Unreleased]
9
+
10
+ ### Fixed
11
+ - Statements without binds failed with `PQsendQuery not allowed in pipeline mode`.
12
+ - Reading pipeline results ignored the per-statement `nil` terminators, so batches of more than one statement failed.
13
+ - Inserts were missing `RETURNING "id"`, so created records had no primary key.
14
+ - Writes hung forever when the worker was not started, had stopped, or the process had forked.
15
+ - Worker crashes or connection failures could leave callers waiting forever; they are now always released.
16
+ - Writes inside a materialized transaction escaped it (and could deadlock on row locks); they now run on the caller's connection.
17
+ - Database errors are translated into the usual ActiveRecord classes (`RecordNotUnique`, `NotNullViolation`, ...).
18
+ - Worker connections are configured like the application's (search path, timezone, variables) and follow the model's database in multi-database setups.
19
+ - Nested `ActivePipeline.bypass` blocks restore the outer state.
20
+
21
+ ### Added
22
+ - `max_batch_size` and `wait_timeout` settings.
23
+ - `batch.active_pipeline` instrumentation event; pipelined statements are logged as `sql.active_record`.
24
+ - `stop!` drains queued statements before shutting down.
25
+
26
+ ### Changed
27
+ - Requires Ruby >= 3.2 and ActiveRecord >= 7.1.
28
+
29
+ ## [0.1.0]
30
+
31
+ - Initial release.
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2026 Carlos (Thrashattack) Cunha
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,189 @@
1
+ # ActivePipeline
2
+
3
+ [![CI](https://github.com/thrashattack/active_pipeline/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/thrashattack/active_pipeline/actions/workflows/ci.yml)
4
+ [![Coverage Status](https://coveralls.io/repos/github/Thrashattack/active_pipeline/badge.svg?branch=main)](https://coveralls.io/github/thrashattack/active_pipeline?branch=main)
5
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE.txt)
6
+
7
+ Transparent, Go-channel-like write batching for ActiveRecord on PostgreSQL.
8
+
9
+ ActivePipeline intercepts the `INSERT`, `UPDATE` and `DELETE` statements issued by your
10
+ threads (Sidekiq workers, multi-threaded Puma), buffers them for a short window and flushes
11
+ them per table in a single round trip using
12
+ [PostgreSQL pipeline mode](https://www.postgresql.org/docs/current/libpq-pipeline-mode.html).
13
+ Each caller thread blocks until its own statement is resolved, so application code
14
+ keeps the usual synchronous API: `create!` still returns a persisted record with its id,
15
+ and `update_all` still returns the affected row count.
16
+
17
+ If one statement in a batch fails, only that caller gets the error. The rest of the batch
18
+ is retried and committed.
19
+
20
+ ## Table of contents
21
+
22
+ - [How it works](#how-it-works)
23
+ - [Requirements](#requirements)
24
+ - [Installation](#installation)
25
+ - [Usage](#usage)
26
+ - [Configuration](#configuration)
27
+ - [Transactions and tradeoffs](#transactions-and-tradeoffs)
28
+ - [Instrumentation](#instrumentation)
29
+ - [Development](#development)
30
+ - [Contributing](#contributing)
31
+ - [License](#license)
32
+
33
+ ## How it works
34
+
35
+ ```
36
+ Sidekiq threads ActivePipeline worker PostgreSQL
37
+ ─────────────── ───────────────────── ──────────
38
+ User.create! ─┐ ┌─ group "users" ─ pipeline ─────────▶ BEGIN (implicit)
39
+ User.create! ─┼─▶ queue ─ window ──┤ INSERT, INSERT, UPDATE ... ...
40
+ Post.update! ─┘ (IVar per call) └─ group "posts" ─ pipeline ─────────▶ SYNC / COMMIT
41
+ ▲ │
42
+ └──────────── results / errors ───────────┘
43
+ ```
44
+
45
+ 1. A write on a PostgreSQL connection is wrapped in a payload and pushed onto a queue. The
46
+ calling thread then waits on a `Concurrent::IVar`.
47
+ 2. A background worker takes the first statement, keeps collecting for `batch_window_ms`
48
+ (or until `max_batch_size`), and groups the batch by database and table.
49
+ 3. Each group is sent through its own dedicated connection in pipeline mode as one implicit
50
+ transaction. Groups run in parallel.
51
+ 4. If a statement fails, PostgreSQL rolls the whole group back. ActivePipeline rejects
52
+ the failing statement and pipelines the remaining ones again.
53
+ 5. Callers wake up with the same values and the same exception classes
54
+ (`ActiveRecord::RecordNotUnique`, `ActiveRecord::NotNullViolation`, ...) that ActiveRecord
55
+ would have produced.
56
+
57
+ ## Requirements
58
+
59
+ - Ruby >= 3.2
60
+ - ActiveRecord >= 7.1 with the PostgreSQL adapter
61
+ - `pg` >= 1.3, built against libpq >= 14 (pipeline mode)
62
+ - PostgreSQL >= 14
63
+
64
+ ## Installation
65
+
66
+ Add the gem to your application's Gemfile:
67
+
68
+ ```ruby
69
+ gem "active_pipeline"
70
+ ```
71
+
72
+ Then run:
73
+
74
+ ```sh
75
+ bundle install
76
+ ```
77
+
78
+ ## Usage
79
+
80
+ The pipeline only diverts writes while the worker is running. Start it in the processes
81
+ where batching pays off. Sidekiq is the recommended place:
82
+
83
+ ```ruby
84
+ # config/initializers/sidekiq.rb
85
+ Sidekiq.configure_server do |config|
86
+ config.on(:startup) { ActivePipeline.start! }
87
+ config.on(:shutdown) { ActivePipeline.stop! }
88
+ end
89
+ ```
90
+
91
+ In processes that never call `start!` (web servers, consoles, rake tasks), every write goes
92
+ straight through ActiveRecord as usual.
93
+
94
+ `stop!` drains the statements that are already queued before closing its connections. If
95
+ the process forks after `start!`, call `start!` again in the child; the child never reuses
96
+ the parent's worker.
97
+
98
+ To force a block of code to skip the pipeline:
99
+
100
+ ```ruby
101
+ ActivePipeline.bypass do
102
+ Order.transaction do
103
+ order.save!
104
+ order.line_items.each(&:save!)
105
+ end
106
+ end
107
+ ```
108
+
109
+ ## Configuration
110
+
111
+ ```ruby
112
+ # config/initializers/active_pipeline.rb
113
+ ActivePipeline.configure do |config|
114
+ config.enabled = true # master switch
115
+ config.batch_window_ms = 50 # how long to keep collecting after the first statement arrives
116
+ config.max_batch_size = 500 # statements per flush
117
+ config.wait_timeout = nil # seconds a caller waits before ActivePipeline::TimeoutError (nil = forever)
118
+ end
119
+ ```
120
+
121
+ | Option | Default | Description |
122
+ |-------------------|---------|-------------|
123
+ | `enabled` | `true` | When `false`, all writes use the regular ActiveRecord path. |
124
+ | `batch_window_ms` | `50` | Maximum latency added to a write while the worker collects a batch. |
125
+ | `max_batch_size` | `500` | Upper bound of statements flushed in one pipeline. |
126
+ | `wait_timeout` | `nil` | Raises `ActivePipeline::TimeoutError` in the caller after this many seconds. The statement is **not** cancelled and may still commit afterwards. |
127
+
128
+ ## Transactions and tradeoffs
129
+
130
+ Pipelined statements run on the worker's connections, not on the connection of the calling
131
+ thread. Keep the following in mind:
132
+
133
+ - **Materialized transactions are respected.** If the current connection already started a
134
+ transaction (it ran any query inside `transaction do ... end`, including the `SELECT` of a
135
+ uniqueness validation or a `lock`), the write runs on that connection as usual. This keeps
136
+ rollbacks and row locks correct and avoids deadlocking against your own transaction.
137
+ - **Lazy transactions are not.** A `transaction` block whose first statement is a write
138
+ gets that write committed by the pipeline independently. A later `raise
139
+ ActiveRecord::Rollback` will not undo it. Use `ActivePipeline.bypass` or read something
140
+ first when several writes must commit atomically.
141
+ - **Connections.** The worker opens one connection per database and table it writes to.
142
+ Size `max_connections` on the server accordingly.
143
+ - **Ordering.** Statements are applied in queue order within a table. There is no ordering
144
+ guarantee between different tables in the same batch. A single thread is never affected,
145
+ because each caller waits for its own statement to finish.
146
+ - **Bulk `insert_all`/`upsert_all`** and raw `execute` calls are not pipelined.
147
+
148
+ ## Instrumentation
149
+
150
+ Pipelined statements are still logged and instrumented as `sql.active_record`. In addition,
151
+ the worker emits one `batch.active_pipeline` event per flushed group:
152
+
153
+ ```ruby
154
+ ActiveSupport::Notifications.subscribe("batch.active_pipeline") do |event|
155
+ event.payload # => { table: "users", size: 42, rounds: 2, failed: 1 }
156
+ end
157
+ ```
158
+
159
+ `rounds` is the number of pipelines needed to isolate failures. `failed` is the number of
160
+ statements that were rejected.
161
+
162
+ ## Development
163
+
164
+ You need a running PostgreSQL. The specs connect using `DB_HOST`, `DB_PORT`, `DB_NAME`,
165
+ `DB_USER` and `DB_PASSWORD`. The defaults are `localhost`, `5432`, `active_pipeline_test`,
166
+ `postgres` and `postgres`.
167
+
168
+ ```sh
169
+ createdb active_pipeline_test
170
+ bundle install
171
+ bundle exec rake # runs the specs
172
+ ```
173
+
174
+ Every run writes a coverage report to `coverage/index.html`. On CI it also produces
175
+ `coverage/lcov.info`, which is uploaded to Coveralls.
176
+
177
+ ## Contributing
178
+
179
+ Bug reports and pull requests are welcome on GitHub at
180
+ https://github.com/thrashattack/active_pipeline.
181
+
182
+ 1. Fork the repository
183
+ 2. Create your feature branch (`git checkout -b my-feature`)
184
+ 3. Add specs for your change and make sure `bundle exec rake` passes
185
+ 4. Open a pull request
186
+
187
+ ## License
188
+
189
+ The gem is available as open source under the terms of the [MIT License](LICENSE.txt).
@@ -0,0 +1,52 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ActivePipeline
4
+ # Prepended to ActiveRecord::ConnectionAdapters::PostgreSQLAdapter. Diverts
5
+ # INSERT/UPDATE/DELETE statements to the worker when it is safe to do so and
6
+ # falls back to the stock implementation otherwise.
7
+ module AdapterExtension
8
+ def exec_insert(sql, name = nil, binds = [], pk = nil, sequence_name = nil, returning: nil)
9
+ return super unless pipeline_write? && (use_insert_returning? || pk == false)
10
+
11
+ insert_sql, insert_binds = sql_for_insert(sql, pk, binds, returning)
12
+ pipeline_execute(:insert, insert_sql, name, insert_binds) { super }
13
+ end
14
+
15
+ def exec_update(sql, name = nil, binds = [])
16
+ return super unless pipeline_write?
17
+
18
+ pipeline_execute(:update, sql, name, binds) { super }
19
+ end
20
+
21
+ def exec_delete(sql, name = nil, binds = [])
22
+ return super unless pipeline_write?
23
+
24
+ pipeline_execute(:delete, sql, name, binds) { super }
25
+ end
26
+
27
+ private
28
+
29
+ # A materialized transaction means this connection already sent BEGIN (and
30
+ # may hold row locks), so the statement has to run on this very connection:
31
+ # pipelining it would escape the transaction or deadlock against it.
32
+ def pipeline_write?
33
+ ActivePipeline.enabled &&
34
+ ActivePipeline.running? &&
35
+ !ActivePipeline.bypassed? &&
36
+ !current_transaction.materialized?
37
+ end
38
+
39
+ def pipeline_execute(operation, sql, name, binds)
40
+ casted_binds = type_casted_binds(binds)
41
+ payload = Payload.new(operation: operation, sql: sql, binds: casted_binds, db_config: pool.db_config)
42
+
43
+ return yield unless ActivePipeline.enqueue(payload)
44
+
45
+ log(sql, name, binds, casted_binds) do
46
+ payload.wait(ActivePipeline.wait_timeout)
47
+ rescue PG::Error => e
48
+ raise translate_exception_class(e, sql, binds)
49
+ end
50
+ end
51
+ end
52
+ end
@@ -0,0 +1,55 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ActivePipeline
4
+ # A single write statement waiting in the queue. The caller thread blocks on
5
+ # +wait+ until the worker resolves it with +fulfill+ or +reject+.
6
+ class Payload
7
+ UNKNOWN_TABLE = "unknown"
8
+
9
+ IDENTIFIER = /(?:"(?:[^"]|"")+"|[\w$]+)/
10
+ TABLE_REGEX = %r{
11
+ \A\s*(?:/\*.*?\*/\s*)* # leading query-log comments
12
+ (?:INSERT\s+INTO|UPDATE|DELETE\s+FROM)\s+
13
+ (#{IDENTIFIER}(?:\.#{IDENTIFIER})?) # optionally schema-qualified
14
+ }mix
15
+
16
+ attr_reader :operation, :sql, :binds, :db_config, :table_name
17
+
18
+ def self.table_name_for(sql)
19
+ match = TABLE_REGEX.match(sql)
20
+ match && match[1].delete('"')
21
+ end
22
+
23
+ def initialize(operation:, sql:, binds:, db_config:)
24
+ @operation = operation
25
+ @sql = sql
26
+ @binds = binds
27
+ @db_config = db_config
28
+ @table_name = self.class.table_name_for(sql) || UNKNOWN_TABLE
29
+ @ivar = Concurrent::IVar.new
30
+ end
31
+
32
+ def group_key
33
+ [db_config, table_name]
34
+ end
35
+
36
+ def fulfill(value)
37
+ @ivar.set(value) unless @ivar.complete?
38
+ end
39
+
40
+ def reject(error)
41
+ @ivar.fail(error) unless @ivar.complete?
42
+ end
43
+
44
+ def resolved?
45
+ @ivar.complete?
46
+ end
47
+
48
+ def wait(timeout = nil)
49
+ value = @ivar.value!(timeout)
50
+ raise TimeoutError, "statement was not resolved within #{timeout}s" unless @ivar.complete?
51
+
52
+ value
53
+ end
54
+ end
55
+ end
@@ -0,0 +1,3 @@
1
+ module ActivePipeline
2
+ VERSION = "0.1.0"
3
+ end
@@ -0,0 +1,233 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ActivePipeline
4
+ # Background thread that drains the queue, groups statements per
5
+ # (database, table) and flushes each group through a dedicated connection in
6
+ # PostgreSQL pipeline mode.
7
+ #
8
+ # Every group runs as one implicit transaction: when a statement fails the
9
+ # server rolls the whole group back, the failing statement is rejected and
10
+ # the remaining ones are pipelined again.
11
+ class Worker
12
+ ABORTED = Object.new.freeze
13
+ private_constant :ABORTED
14
+
15
+ def initialize
16
+ @queue = ::Queue.new
17
+ @adapters = Concurrent::Map.new
18
+ @thread = nil
19
+ @pid = nil
20
+ end
21
+
22
+ def start!
23
+ return if @thread
24
+
25
+ @pid = Process.pid
26
+ @thread = Thread.new { run_loop }
27
+ @thread.name = "active_pipeline"
28
+ end
29
+
30
+ # Closes the queue, lets the thread drain whatever is already queued and
31
+ # releases the connections.
32
+ def stop!
33
+ # A forked child has no worker thread and shares the parent's sockets:
34
+ # closing them would kill the parent's sessions, so it is left untouched.
35
+ return if @thread.nil? || forked?
36
+
37
+ @queue.close
38
+ join_thread
39
+ reject_leftovers(Error.new("ActivePipeline worker stopped"))
40
+ end
41
+
42
+ def running?
43
+ !@queue.closed? && !forked? && !!@thread&.alive?
44
+ end
45
+
46
+ def enqueue(payload)
47
+ return false unless running?
48
+
49
+ @queue.push(payload)
50
+ true
51
+ rescue ClosedQueueError
52
+ false
53
+ end
54
+
55
+ private
56
+
57
+ # Thread#join re-raises a crash that the thread already reported.
58
+ def join_thread
59
+ @thread.join
60
+ rescue StandardError
61
+ nil
62
+ end
63
+
64
+ def forked?
65
+ @pid != Process.pid
66
+ end
67
+
68
+ def run_loop
69
+ while (first = @queue.pop)
70
+ batch = [first]
71
+ collect_batch(batch)
72
+ process_batch(batch)
73
+ end
74
+ rescue Exception => e # rubocop:disable Lint/RescueException
75
+ # The thread is dying: release every caller instead of leaving it parked.
76
+ # running? turns false, so later writes fall back to regular ActiveRecord.
77
+ batch&.each { |payload| payload.reject(e) }
78
+ reject_leftovers(e)
79
+ raise
80
+ ensure
81
+ disconnect_all
82
+ end
83
+
84
+ def collect_batch(batch)
85
+ deadline = monotonic_now + (ActivePipeline.batch_window_ms / 1000.0)
86
+
87
+ while batch.size < ActivePipeline.max_batch_size
88
+ remaining = deadline - monotonic_now
89
+ break if remaining <= 0
90
+
91
+ payload = @queue.pop(timeout: remaining)
92
+ break unless payload
93
+
94
+ batch << payload
95
+ end
96
+ end
97
+
98
+ def process_batch(batch)
99
+ futures = batch.group_by(&:group_key).map do |key, items|
100
+ Concurrent::Promises.future(key, items) { |k, i| process_group(k, i) }
101
+ end
102
+ futures.each(&:wait)
103
+ ensure
104
+ batch.each { |payload| payload.reject(Error.new("statement was not processed")) }
105
+ end
106
+
107
+ def process_group(key, items)
108
+ resolutions = {}.compare_by_identity
109
+
110
+ ActiveSupport::Notifications.instrument(BATCH_EVENT, table: key.last, size: items.size) do |event|
111
+ run_isolated(key, items, resolutions, event)
112
+ end
113
+ rescue StandardError => e
114
+ discard_adapter(key)
115
+ items.each { |item| resolutions[item] ||= [:reject, e] }
116
+ ensure
117
+ # Resolve only after the instrumentation finished so subscribers always
118
+ # observe the batch before the callers wake up.
119
+ resolutions.each { |item, (action, value)| item.public_send(action, value) }
120
+ end
121
+
122
+ def run_isolated(key, items, resolutions, event)
123
+ pending = items.dup
124
+ rounds = 0
125
+ failed = 0
126
+
127
+ until pending.empty?
128
+ rounds += 1
129
+ results = execute_pipeline(raw_connection_for(key), pending)
130
+ index = results.index { |result| result.is_a?(Exception) }
131
+
132
+ if index
133
+ failed += 1
134
+ resolutions[pending.delete_at(index)] = [:reject, results[index]]
135
+ elsif results.any?(ABORTED)
136
+ raise Error, "pipeline aborted without reporting an error"
137
+ else
138
+ pending.zip(results) { |item, result| resolutions[item] = [:fulfill, result] }
139
+ pending = []
140
+ end
141
+ end
142
+ ensure
143
+ event[:rounds] = rounds
144
+ event[:failed] = failed
145
+ end
146
+
147
+ def execute_pipeline(conn, items)
148
+ conn.enter_pipeline_mode
149
+ items.each { |item| conn.send_query_params(item.sql, item.binds) }
150
+ conn.pipeline_sync
151
+
152
+ results = items.map { |item| convert(read_result(conn), item) }
153
+ read_sync(conn)
154
+ conn.exit_pipeline_mode
155
+ results
156
+ end
157
+
158
+ # In pipeline mode each statement's results are terminated by a nil.
159
+ def read_result(conn)
160
+ last = nil
161
+ while (result = conn.get_result)
162
+ last = result
163
+ end
164
+ last
165
+ end
166
+
167
+ def convert(result, item)
168
+ case result&.result_status
169
+ when PG::PGRES_TUPLES_OK, PG::PGRES_COMMAND_OK
170
+ value = item.operation == :insert ? ActiveRecord::Result.new(result.fields, result.values) : result.cmd_tuples
171
+ result.clear
172
+ value
173
+ when PG::PGRES_PIPELINE_ABORTED
174
+ ABORTED
175
+ when PG::PGRES_FATAL_ERROR
176
+ error_for(result)
177
+ else
178
+ raise Error, "unexpected result status #{result&.result_status.inspect} for #{item.sql}"
179
+ end
180
+ end
181
+
182
+ # PG::Result#check raises the specific PG::Error subclass (UniqueViolation,
183
+ # NotNullViolation...) with the result attached, which is what ActiveRecord
184
+ # needs to translate it into RecordNotUnique, NotNullViolation, etc.
185
+ def error_for(result)
186
+ result.check
187
+ Error.new("statement failed without an error message")
188
+ rescue PG::Error => e
189
+ e
190
+ end
191
+
192
+ # The implicit transaction commits at the sync point. An error there (e.g. a
193
+ # deferred constraint) can't be attributed to a statement, so it is raised
194
+ # and rejects the whole group.
195
+ def read_sync(conn)
196
+ result = conn.get_result
197
+ return if result&.result_status == PG::PGRES_PIPELINE_SYNC
198
+
199
+ result&.check
200
+ raise Error, "pipeline sync was not acknowledged"
201
+ end
202
+
203
+ # Each group gets its own ActiveRecord adapter so the session is configured
204
+ # exactly like the application's (search_path, timezone, variables...).
205
+ def raw_connection_for(key)
206
+ adapter = @adapters.compute_if_absent(key) do
207
+ ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.new(key.first.configuration_hash)
208
+ end
209
+ adapter.raw_connection
210
+ end
211
+
212
+ def discard_adapter(key)
213
+ adapter = @adapters.delete(key)
214
+ adapter&.disconnect!
215
+ rescue StandardError
216
+ nil
217
+ end
218
+
219
+ def disconnect_all
220
+ @adapters.each_key { |key| discard_adapter(key) }
221
+ end
222
+
223
+ def reject_leftovers(error)
224
+ while (payload = @queue.pop(timeout: 0))
225
+ payload.reject(error)
226
+ end
227
+ end
228
+
229
+ def monotonic_now
230
+ Process.clock_gettime(Process::CLOCK_MONOTONIC)
231
+ end
232
+ end
233
+ end
@@ -0,0 +1,87 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "concurrent"
4
+ require "active_record"
5
+ require "pg"
6
+
7
+ require_relative "active_pipeline/version"
8
+ require_relative "active_pipeline/payload"
9
+ require_relative "active_pipeline/worker"
10
+ require_relative "active_pipeline/adapter_extension"
11
+
12
+ module ActivePipeline
13
+ class Error < StandardError; end
14
+
15
+ # Raised in the caller thread when +wait_timeout+ elapses before the worker
16
+ # resolved the statement. The statement may still be committed afterwards.
17
+ class TimeoutError < Error; end
18
+
19
+ # ActiveSupport::Notifications event emitted once per (database, table) group
20
+ # flushed by the worker. Payload: :table, :size, :rounds, :failed.
21
+ BATCH_EVENT = "batch.active_pipeline"
22
+
23
+ @mutex = Mutex.new
24
+
25
+ class << self
26
+ # Master switch. When false every write goes straight through ActiveRecord.
27
+ attr_accessor :enabled
28
+
29
+ # How long the worker keeps collecting statements after the first one arrives.
30
+ attr_accessor :batch_window_ms
31
+
32
+ # Upper bound of statements flushed in a single pipeline.
33
+ attr_accessor :max_batch_size
34
+
35
+ # Seconds a caller waits for its statement to be resolved (nil waits forever).
36
+ attr_accessor :wait_timeout
37
+
38
+ def configure
39
+ yield self
40
+ end
41
+
42
+ def start!
43
+ @mutex.synchronize do
44
+ @worker = Worker.new.tap(&:start!) unless @worker&.running?
45
+ end
46
+ end
47
+
48
+ def stop!
49
+ @mutex.synchronize do
50
+ @worker&.stop!
51
+ @worker = nil
52
+ end
53
+ end
54
+
55
+ def running?
56
+ !!@worker&.running?
57
+ end
58
+
59
+ # Runs the block with the pipeline disabled for the current thread/fiber.
60
+ def bypass
61
+ previous = Thread.current[:active_pipeline_bypass]
62
+ Thread.current[:active_pipeline_bypass] = true
63
+ yield
64
+ ensure
65
+ Thread.current[:active_pipeline_bypass] = previous
66
+ end
67
+
68
+ def bypassed?
69
+ !!Thread.current[:active_pipeline_bypass]
70
+ end
71
+
72
+ # Hands a payload to the running worker. Returns false when nothing is
73
+ # running so the caller can fall back to a regular, synchronous write.
74
+ def enqueue(payload) # :nodoc:
75
+ !!@worker&.enqueue(payload)
76
+ end
77
+ end
78
+
79
+ self.enabled = true
80
+ self.batch_window_ms = 50
81
+ self.max_batch_size = 500
82
+ self.wait_timeout = nil
83
+ end
84
+
85
+ ActiveSupport.on_load(:active_record_postgresqladapter) do
86
+ prepend ActivePipeline::AdapterExtension
87
+ end
metadata ADDED
@@ -0,0 +1,96 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: active_pipeline
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - Carlos (Thrashattack) Cunha
8
+ bindir: bin
9
+ cert_chain: []
10
+ date: 1980-01-02 00:00:00.000000000 Z
11
+ dependencies:
12
+ - !ruby/object:Gem::Dependency
13
+ name: activerecord
14
+ requirement: !ruby/object:Gem::Requirement
15
+ requirements:
16
+ - - ">="
17
+ - !ruby/object:Gem::Version
18
+ version: '7.1'
19
+ type: :runtime
20
+ prerelease: false
21
+ version_requirements: !ruby/object:Gem::Requirement
22
+ requirements:
23
+ - - ">="
24
+ - !ruby/object:Gem::Version
25
+ version: '7.1'
26
+ - !ruby/object:Gem::Dependency
27
+ name: concurrent-ruby
28
+ requirement: !ruby/object:Gem::Requirement
29
+ requirements:
30
+ - - "~>"
31
+ - !ruby/object:Gem::Version
32
+ version: '1.1'
33
+ type: :runtime
34
+ prerelease: false
35
+ version_requirements: !ruby/object:Gem::Requirement
36
+ requirements:
37
+ - - "~>"
38
+ - !ruby/object:Gem::Version
39
+ version: '1.1'
40
+ - !ruby/object:Gem::Dependency
41
+ name: pg
42
+ requirement: !ruby/object:Gem::Requirement
43
+ requirements:
44
+ - - ">="
45
+ - !ruby/object:Gem::Version
46
+ version: '1.3'
47
+ type: :runtime
48
+ prerelease: false
49
+ version_requirements: !ruby/object:Gem::Requirement
50
+ requirements:
51
+ - - ">="
52
+ - !ruby/object:Gem::Version
53
+ version: '1.3'
54
+ description: Implements Go-channel-like write batching for Rails and Sidekiq using
55
+ PostgreSQL pipeline mode, with per-statement error isolation.
56
+ email:
57
+ - henrique2092@gmail.com
58
+ executables: []
59
+ extensions: []
60
+ extra_rdoc_files: []
61
+ files:
62
+ - CHANGELOG.md
63
+ - LICENSE.txt
64
+ - README.md
65
+ - lib/active_pipeline.rb
66
+ - lib/active_pipeline/adapter_extension.rb
67
+ - lib/active_pipeline/payload.rb
68
+ - lib/active_pipeline/version.rb
69
+ - lib/active_pipeline/worker.rb
70
+ homepage: https://github.com/thrashattack/active_pipeline
71
+ licenses:
72
+ - MIT
73
+ metadata:
74
+ homepage_uri: https://github.com/thrashattack/active_pipeline
75
+ source_code_uri: https://github.com/thrashattack/active_pipeline
76
+ changelog_uri: https://github.com/thrashattack/active_pipeline/blob/main/CHANGELOG.md
77
+ bug_tracker_uri: https://github.com/thrashattack/active_pipeline/issues
78
+ rubygems_mfa_required: 'true'
79
+ rdoc_options: []
80
+ require_paths:
81
+ - lib
82
+ required_ruby_version: !ruby/object:Gem::Requirement
83
+ requirements:
84
+ - - ">="
85
+ - !ruby/object:Gem::Version
86
+ version: '3.2'
87
+ required_rubygems_version: !ruby/object:Gem::Requirement
88
+ requirements:
89
+ - - ">="
90
+ - !ruby/object:Gem::Version
91
+ version: '0'
92
+ requirements: []
93
+ rubygems_version: 3.6.9
94
+ specification_version: 4
95
+ summary: Transparent PostgreSQL pipeline batching for ActiveRecord.
96
+ test_files: []