active_pipeline 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +31 -0
- data/LICENSE.txt +21 -0
- data/README.md +189 -0
- data/lib/active_pipeline/adapter_extension.rb +52 -0
- data/lib/active_pipeline/payload.rb +55 -0
- data/lib/active_pipeline/version.rb +3 -0
- data/lib/active_pipeline/worker.rb +233 -0
- data/lib/active_pipeline.rb +87 -0
- metadata +96 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 07b59b4a629962f09f8a6fde03fece7b3a4e1bcec69e3bf711657e83b1115360
|
|
4
|
+
data.tar.gz: fb6630f5e422714efae2314896925c2becb8f849f774446520ad7e8ff9ac83bd
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 7a48549372312bfaa19cb0f21a7960ad93d9cb583db50a675263698683c336f1cfe0fcd70b7497ec0e16126ec7822bd39ab126fb14a10b37c3d0cf2d8350100b
|
|
7
|
+
data.tar.gz: d58eb657403ca873f1721ecb9700318abb2fea358a35be516f1980b3542ba905caa8d0009c4dd065c51080be5ea8d09df8b1c1b87fd9cde4ea276af15b586702
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
- Statements without binds failed with `PQsendQuery not allowed in pipeline mode`.
|
|
12
|
+
- Reading pipeline results ignored the per-statement `nil` terminators, so batches of more than one statement failed.
|
|
13
|
+
- Inserts were missing `RETURNING "id"`, so created records had no primary key.
|
|
14
|
+
- Writes hung forever when the worker was not started, had stopped, or the process had forked.
|
|
15
|
+
- Worker crashes or connection failures could leave callers waiting forever; they are now always released.
|
|
16
|
+
- Writes inside a materialized transaction escaped it (and could deadlock on row locks); they now run on the caller's connection.
|
|
17
|
+
- Database errors are translated into the usual ActiveRecord classes (`RecordNotUnique`, `NotNullViolation`, ...).
|
|
18
|
+
- Worker connections are configured like the application's (search path, timezone, variables) and follow the model's database in multi-database setups.
|
|
19
|
+
- Nested `ActivePipeline.bypass` blocks restore the outer state.
|
|
20
|
+
|
|
21
|
+
### Added
|
|
22
|
+
- `max_batch_size` and `wait_timeout` settings.
|
|
23
|
+
- `batch.active_pipeline` instrumentation event; pipelined statements are logged as `sql.active_record`.
|
|
24
|
+
- `stop!` drains queued statements before shutting down.
|
|
25
|
+
|
|
26
|
+
### Changed
|
|
27
|
+
- Requires Ruby >= 3.2 and ActiveRecord >= 7.1.
|
|
28
|
+
|
|
29
|
+
## [0.1.0]
|
|
30
|
+
|
|
31
|
+
- Initial release.
|
data/LICENSE.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
The MIT License (MIT)
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Carlos (Thrashattack) Cunha
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
|
13
|
+
all copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,189 @@
|
|
|
1
|
+
# ActivePipeline
|
|
2
|
+
|
|
3
|
+
[](https://github.com/thrashattack/active_pipeline/actions/workflows/ci.yml)
|
|
4
|
+
[](https://coveralls.io/github/thrashattack/active_pipeline?branch=main)
|
|
5
|
+
[](LICENSE.txt)
|
|
6
|
+
|
|
7
|
+
Transparent, Go-channel-like write batching for ActiveRecord on PostgreSQL.
|
|
8
|
+
|
|
9
|
+
ActivePipeline intercepts the `INSERT`, `UPDATE` and `DELETE` statements issued by your
|
|
10
|
+
threads (Sidekiq workers, multi-threaded Puma), buffers them for a short window and flushes
|
|
11
|
+
them per table in a single round trip using
|
|
12
|
+
[PostgreSQL pipeline mode](https://www.postgresql.org/docs/current/libpq-pipeline-mode.html).
|
|
13
|
+
Each caller thread blocks until its own statement is resolved, so application code
|
|
14
|
+
keeps the usual synchronous API: `create!` still returns a persisted record with its id,
|
|
15
|
+
and `update_all` still returns the affected row count.
|
|
16
|
+
|
|
17
|
+
If one statement in a batch fails, only that caller gets the error. The rest of the batch
|
|
18
|
+
is retried and committed.
|
|
19
|
+
|
|
20
|
+
## Table of contents
|
|
21
|
+
|
|
22
|
+
- [How it works](#how-it-works)
|
|
23
|
+
- [Requirements](#requirements)
|
|
24
|
+
- [Installation](#installation)
|
|
25
|
+
- [Usage](#usage)
|
|
26
|
+
- [Configuration](#configuration)
|
|
27
|
+
- [Transactions and tradeoffs](#transactions-and-tradeoffs)
|
|
28
|
+
- [Instrumentation](#instrumentation)
|
|
29
|
+
- [Development](#development)
|
|
30
|
+
- [Contributing](#contributing)
|
|
31
|
+
- [License](#license)
|
|
32
|
+
|
|
33
|
+
## How it works
|
|
34
|
+
|
|
35
|
+
```
|
|
36
|
+
Sidekiq threads ActivePipeline worker PostgreSQL
|
|
37
|
+
─────────────── ───────────────────── ──────────
|
|
38
|
+
User.create! ─┐ ┌─ group "users" ─ pipeline ─────────▶ BEGIN (implicit)
|
|
39
|
+
User.create! ─┼─▶ queue ─ window ──┤ INSERT, INSERT, UPDATE ... ...
|
|
40
|
+
Post.update! ─┘ (IVar per call) └─ group "posts" ─ pipeline ─────────▶ SYNC / COMMIT
|
|
41
|
+
▲ │
|
|
42
|
+
└──────────── results / errors ───────────┘
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
1. A write on a PostgreSQL connection is wrapped in a payload and pushed onto a queue. The
|
|
46
|
+
calling thread then waits on a `Concurrent::IVar`.
|
|
47
|
+
2. A background worker takes the first statement, keeps collecting for `batch_window_ms`
|
|
48
|
+
(or until `max_batch_size`), and groups the batch by database and table.
|
|
49
|
+
3. Each group is sent through its own dedicated connection in pipeline mode as one implicit
|
|
50
|
+
transaction. Groups run in parallel.
|
|
51
|
+
4. If a statement fails, PostgreSQL rolls the whole group back. ActivePipeline rejects
|
|
52
|
+
the failing statement and pipelines the remaining ones again.
|
|
53
|
+
5. Callers wake up with the same values and the same exception classes
|
|
54
|
+
(`ActiveRecord::RecordNotUnique`, `ActiveRecord::NotNullViolation`, ...) that ActiveRecord
|
|
55
|
+
would have produced.
|
|
56
|
+
|
|
57
|
+
## Requirements
|
|
58
|
+
|
|
59
|
+
- Ruby >= 3.2
|
|
60
|
+
- ActiveRecord >= 7.1 with the PostgreSQL adapter
|
|
61
|
+
- `pg` >= 1.3, built against libpq >= 14 (pipeline mode)
|
|
62
|
+
- PostgreSQL >= 14
|
|
63
|
+
|
|
64
|
+
## Installation
|
|
65
|
+
|
|
66
|
+
Add the gem to your application's Gemfile:
|
|
67
|
+
|
|
68
|
+
```ruby
|
|
69
|
+
gem "active_pipeline"
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Then run:
|
|
73
|
+
|
|
74
|
+
```sh
|
|
75
|
+
bundle install
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
## Usage
|
|
79
|
+
|
|
80
|
+
The pipeline only diverts writes while the worker is running. Start it in the processes
|
|
81
|
+
where batching pays off. Sidekiq is the recommended place:
|
|
82
|
+
|
|
83
|
+
```ruby
|
|
84
|
+
# config/initializers/sidekiq.rb
|
|
85
|
+
Sidekiq.configure_server do |config|
|
|
86
|
+
config.on(:startup) { ActivePipeline.start! }
|
|
87
|
+
config.on(:shutdown) { ActivePipeline.stop! }
|
|
88
|
+
end
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
In processes that never call `start!` (web servers, consoles, rake tasks), every write goes
|
|
92
|
+
straight through ActiveRecord as usual.
|
|
93
|
+
|
|
94
|
+
`stop!` drains the statements that are already queued before closing its connections. If
|
|
95
|
+
the process forks after `start!`, call `start!` again in the child; the child never reuses
|
|
96
|
+
the parent's worker.
|
|
97
|
+
|
|
98
|
+
To force a block of code to skip the pipeline:
|
|
99
|
+
|
|
100
|
+
```ruby
|
|
101
|
+
ActivePipeline.bypass do
|
|
102
|
+
Order.transaction do
|
|
103
|
+
order.save!
|
|
104
|
+
order.line_items.each(&:save!)
|
|
105
|
+
end
|
|
106
|
+
end
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
## Configuration
|
|
110
|
+
|
|
111
|
+
```ruby
|
|
112
|
+
# config/initializers/active_pipeline.rb
|
|
113
|
+
ActivePipeline.configure do |config|
|
|
114
|
+
config.enabled = true # master switch
|
|
115
|
+
config.batch_window_ms = 50 # how long to keep collecting after the first statement arrives
|
|
116
|
+
config.max_batch_size = 500 # statements per flush
|
|
117
|
+
config.wait_timeout = nil # seconds a caller waits before ActivePipeline::TimeoutError (nil = forever)
|
|
118
|
+
end
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
| Option | Default | Description |
|
|
122
|
+
|-------------------|---------|-------------|
|
|
123
|
+
| `enabled` | `true` | When `false`, all writes use the regular ActiveRecord path. |
|
|
124
|
+
| `batch_window_ms` | `50` | Maximum latency added to a write while the worker collects a batch. |
|
|
125
|
+
| `max_batch_size` | `500` | Upper bound of statements flushed in one pipeline. |
|
|
126
|
+
| `wait_timeout` | `nil` | Raises `ActivePipeline::TimeoutError` in the caller after this many seconds. The statement is **not** cancelled and may still commit afterwards. |
|
|
127
|
+
|
|
128
|
+
## Transactions and tradeoffs
|
|
129
|
+
|
|
130
|
+
Pipelined statements run on the worker's connections, not on the connection of the calling
|
|
131
|
+
thread. Keep the following in mind:
|
|
132
|
+
|
|
133
|
+
- **Materialized transactions are respected.** If the current connection already started a
|
|
134
|
+
transaction (it ran any query inside `transaction do ... end`, including the `SELECT` of a
|
|
135
|
+
uniqueness validation or a `lock`), the write runs on that connection as usual. This keeps
|
|
136
|
+
rollbacks and row locks correct and avoids deadlocking against your own transaction.
|
|
137
|
+
- **Lazy transactions are not.** A `transaction` block whose first statement is a write
|
|
138
|
+
gets that write committed by the pipeline independently. A later `raise
|
|
139
|
+
ActiveRecord::Rollback` will not undo it. Use `ActivePipeline.bypass` or read something
|
|
140
|
+
first when several writes must commit atomically.
|
|
141
|
+
- **Connections.** The worker opens one connection per database and table it writes to.
|
|
142
|
+
Size `max_connections` on the server accordingly.
|
|
143
|
+
- **Ordering.** Statements are applied in queue order within a table. There is no ordering
|
|
144
|
+
guarantee between different tables in the same batch. A single thread is never affected,
|
|
145
|
+
because each caller waits for its own statement to finish.
|
|
146
|
+
- **Bulk `insert_all`/`upsert_all`** and raw `execute` calls are not pipelined.
|
|
147
|
+
|
|
148
|
+
## Instrumentation
|
|
149
|
+
|
|
150
|
+
Pipelined statements are still logged and instrumented as `sql.active_record`. In addition,
|
|
151
|
+
the worker emits one `batch.active_pipeline` event per flushed group:
|
|
152
|
+
|
|
153
|
+
```ruby
|
|
154
|
+
ActiveSupport::Notifications.subscribe("batch.active_pipeline") do |event|
|
|
155
|
+
event.payload # => { table: "users", size: 42, rounds: 2, failed: 1 }
|
|
156
|
+
end
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
`rounds` is the number of pipelines needed to isolate failures. `failed` is the number of
|
|
160
|
+
statements that were rejected.
|
|
161
|
+
|
|
162
|
+
## Development
|
|
163
|
+
|
|
164
|
+
You need a running PostgreSQL. The specs connect using `DB_HOST`, `DB_PORT`, `DB_NAME`,
|
|
165
|
+
`DB_USER` and `DB_PASSWORD`. The defaults are `localhost`, `5432`, `active_pipeline_test`,
|
|
166
|
+
`postgres` and `postgres`.
|
|
167
|
+
|
|
168
|
+
```sh
|
|
169
|
+
createdb active_pipeline_test
|
|
170
|
+
bundle install
|
|
171
|
+
bundle exec rake # runs the specs
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
Every run writes a coverage report to `coverage/index.html`. On CI it also produces
|
|
175
|
+
`coverage/lcov.info`, which is uploaded to Coveralls.
|
|
176
|
+
|
|
177
|
+
## Contributing
|
|
178
|
+
|
|
179
|
+
Bug reports and pull requests are welcome on GitHub at
|
|
180
|
+
https://github.com/thrashattack/active_pipeline.
|
|
181
|
+
|
|
182
|
+
1. Fork the repository
|
|
183
|
+
2. Create your feature branch (`git checkout -b my-feature`)
|
|
184
|
+
3. Add specs for your change and make sure `bundle exec rake` passes
|
|
185
|
+
4. Open a pull request
|
|
186
|
+
|
|
187
|
+
## License
|
|
188
|
+
|
|
189
|
+
The gem is available as open source under the terms of the [MIT License](LICENSE.txt).
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ActivePipeline
|
|
4
|
+
# Prepended to ActiveRecord::ConnectionAdapters::PostgreSQLAdapter. Diverts
|
|
5
|
+
# INSERT/UPDATE/DELETE statements to the worker when it is safe to do so and
|
|
6
|
+
# falls back to the stock implementation otherwise.
|
|
7
|
+
module AdapterExtension
|
|
8
|
+
def exec_insert(sql, name = nil, binds = [], pk = nil, sequence_name = nil, returning: nil)
|
|
9
|
+
return super unless pipeline_write? && (use_insert_returning? || pk == false)
|
|
10
|
+
|
|
11
|
+
insert_sql, insert_binds = sql_for_insert(sql, pk, binds, returning)
|
|
12
|
+
pipeline_execute(:insert, insert_sql, name, insert_binds) { super }
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
def exec_update(sql, name = nil, binds = [])
|
|
16
|
+
return super unless pipeline_write?
|
|
17
|
+
|
|
18
|
+
pipeline_execute(:update, sql, name, binds) { super }
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
def exec_delete(sql, name = nil, binds = [])
|
|
22
|
+
return super unless pipeline_write?
|
|
23
|
+
|
|
24
|
+
pipeline_execute(:delete, sql, name, binds) { super }
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
private
|
|
28
|
+
|
|
29
|
+
# A materialized transaction means this connection already sent BEGIN (and
|
|
30
|
+
# may hold row locks), so the statement has to run on this very connection:
|
|
31
|
+
# pipelining it would escape the transaction or deadlock against it.
|
|
32
|
+
def pipeline_write?
|
|
33
|
+
ActivePipeline.enabled &&
|
|
34
|
+
ActivePipeline.running? &&
|
|
35
|
+
!ActivePipeline.bypassed? &&
|
|
36
|
+
!current_transaction.materialized?
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
def pipeline_execute(operation, sql, name, binds)
|
|
40
|
+
casted_binds = type_casted_binds(binds)
|
|
41
|
+
payload = Payload.new(operation: operation, sql: sql, binds: casted_binds, db_config: pool.db_config)
|
|
42
|
+
|
|
43
|
+
return yield unless ActivePipeline.enqueue(payload)
|
|
44
|
+
|
|
45
|
+
log(sql, name, binds, casted_binds) do
|
|
46
|
+
payload.wait(ActivePipeline.wait_timeout)
|
|
47
|
+
rescue PG::Error => e
|
|
48
|
+
raise translate_exception_class(e, sql, binds)
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
end
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ActivePipeline
|
|
4
|
+
# A single write statement waiting in the queue. The caller thread blocks on
|
|
5
|
+
# +wait+ until the worker resolves it with +fulfill+ or +reject+.
|
|
6
|
+
class Payload
|
|
7
|
+
UNKNOWN_TABLE = "unknown"
|
|
8
|
+
|
|
9
|
+
IDENTIFIER = /(?:"(?:[^"]|"")+"|[\w$]+)/
|
|
10
|
+
TABLE_REGEX = %r{
|
|
11
|
+
\A\s*(?:/\*.*?\*/\s*)* # leading query-log comments
|
|
12
|
+
(?:INSERT\s+INTO|UPDATE|DELETE\s+FROM)\s+
|
|
13
|
+
(#{IDENTIFIER}(?:\.#{IDENTIFIER})?) # optionally schema-qualified
|
|
14
|
+
}mix
|
|
15
|
+
|
|
16
|
+
attr_reader :operation, :sql, :binds, :db_config, :table_name
|
|
17
|
+
|
|
18
|
+
def self.table_name_for(sql)
|
|
19
|
+
match = TABLE_REGEX.match(sql)
|
|
20
|
+
match && match[1].delete('"')
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def initialize(operation:, sql:, binds:, db_config:)
|
|
24
|
+
@operation = operation
|
|
25
|
+
@sql = sql
|
|
26
|
+
@binds = binds
|
|
27
|
+
@db_config = db_config
|
|
28
|
+
@table_name = self.class.table_name_for(sql) || UNKNOWN_TABLE
|
|
29
|
+
@ivar = Concurrent::IVar.new
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def group_key
|
|
33
|
+
[db_config, table_name]
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def fulfill(value)
|
|
37
|
+
@ivar.set(value) unless @ivar.complete?
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
def reject(error)
|
|
41
|
+
@ivar.fail(error) unless @ivar.complete?
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
def resolved?
|
|
45
|
+
@ivar.complete?
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def wait(timeout = nil)
|
|
49
|
+
value = @ivar.value!(timeout)
|
|
50
|
+
raise TimeoutError, "statement was not resolved within #{timeout}s" unless @ivar.complete?
|
|
51
|
+
|
|
52
|
+
value
|
|
53
|
+
end
|
|
54
|
+
end
|
|
55
|
+
end
|
|
@@ -0,0 +1,233 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ActivePipeline
|
|
4
|
+
# Background thread that drains the queue, groups statements per
|
|
5
|
+
# (database, table) and flushes each group through a dedicated connection in
|
|
6
|
+
# PostgreSQL pipeline mode.
|
|
7
|
+
#
|
|
8
|
+
# Every group runs as one implicit transaction: when a statement fails the
|
|
9
|
+
# server rolls the whole group back, the failing statement is rejected and
|
|
10
|
+
# the remaining ones are pipelined again.
|
|
11
|
+
class Worker
|
|
12
|
+
ABORTED = Object.new.freeze
|
|
13
|
+
private_constant :ABORTED
|
|
14
|
+
|
|
15
|
+
def initialize
|
|
16
|
+
@queue = ::Queue.new
|
|
17
|
+
@adapters = Concurrent::Map.new
|
|
18
|
+
@thread = nil
|
|
19
|
+
@pid = nil
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def start!
|
|
23
|
+
return if @thread
|
|
24
|
+
|
|
25
|
+
@pid = Process.pid
|
|
26
|
+
@thread = Thread.new { run_loop }
|
|
27
|
+
@thread.name = "active_pipeline"
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
# Closes the queue, lets the thread drain whatever is already queued and
|
|
31
|
+
# releases the connections.
|
|
32
|
+
def stop!
|
|
33
|
+
# A forked child has no worker thread and shares the parent's sockets:
|
|
34
|
+
# closing them would kill the parent's sessions, so it is left untouched.
|
|
35
|
+
return if @thread.nil? || forked?
|
|
36
|
+
|
|
37
|
+
@queue.close
|
|
38
|
+
join_thread
|
|
39
|
+
reject_leftovers(Error.new("ActivePipeline worker stopped"))
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def running?
|
|
43
|
+
!@queue.closed? && !forked? && !!@thread&.alive?
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def enqueue(payload)
|
|
47
|
+
return false unless running?
|
|
48
|
+
|
|
49
|
+
@queue.push(payload)
|
|
50
|
+
true
|
|
51
|
+
rescue ClosedQueueError
|
|
52
|
+
false
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
private
|
|
56
|
+
|
|
57
|
+
# Thread#join re-raises a crash that the thread already reported.
|
|
58
|
+
def join_thread
|
|
59
|
+
@thread.join
|
|
60
|
+
rescue StandardError
|
|
61
|
+
nil
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
def forked?
|
|
65
|
+
@pid != Process.pid
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
def run_loop
|
|
69
|
+
while (first = @queue.pop)
|
|
70
|
+
batch = [first]
|
|
71
|
+
collect_batch(batch)
|
|
72
|
+
process_batch(batch)
|
|
73
|
+
end
|
|
74
|
+
rescue Exception => e # rubocop:disable Lint/RescueException
|
|
75
|
+
# The thread is dying: release every caller instead of leaving it parked.
|
|
76
|
+
# running? turns false, so later writes fall back to regular ActiveRecord.
|
|
77
|
+
batch&.each { |payload| payload.reject(e) }
|
|
78
|
+
reject_leftovers(e)
|
|
79
|
+
raise
|
|
80
|
+
ensure
|
|
81
|
+
disconnect_all
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
def collect_batch(batch)
|
|
85
|
+
deadline = monotonic_now + (ActivePipeline.batch_window_ms / 1000.0)
|
|
86
|
+
|
|
87
|
+
while batch.size < ActivePipeline.max_batch_size
|
|
88
|
+
remaining = deadline - monotonic_now
|
|
89
|
+
break if remaining <= 0
|
|
90
|
+
|
|
91
|
+
payload = @queue.pop(timeout: remaining)
|
|
92
|
+
break unless payload
|
|
93
|
+
|
|
94
|
+
batch << payload
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
def process_batch(batch)
|
|
99
|
+
futures = batch.group_by(&:group_key).map do |key, items|
|
|
100
|
+
Concurrent::Promises.future(key, items) { |k, i| process_group(k, i) }
|
|
101
|
+
end
|
|
102
|
+
futures.each(&:wait)
|
|
103
|
+
ensure
|
|
104
|
+
batch.each { |payload| payload.reject(Error.new("statement was not processed")) }
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
def process_group(key, items)
|
|
108
|
+
resolutions = {}.compare_by_identity
|
|
109
|
+
|
|
110
|
+
ActiveSupport::Notifications.instrument(BATCH_EVENT, table: key.last, size: items.size) do |event|
|
|
111
|
+
run_isolated(key, items, resolutions, event)
|
|
112
|
+
end
|
|
113
|
+
rescue StandardError => e
|
|
114
|
+
discard_adapter(key)
|
|
115
|
+
items.each { |item| resolutions[item] ||= [:reject, e] }
|
|
116
|
+
ensure
|
|
117
|
+
# Resolve only after the instrumentation finished so subscribers always
|
|
118
|
+
# observe the batch before the callers wake up.
|
|
119
|
+
resolutions.each { |item, (action, value)| item.public_send(action, value) }
|
|
120
|
+
end
|
|
121
|
+
|
|
122
|
+
def run_isolated(key, items, resolutions, event)
|
|
123
|
+
pending = items.dup
|
|
124
|
+
rounds = 0
|
|
125
|
+
failed = 0
|
|
126
|
+
|
|
127
|
+
until pending.empty?
|
|
128
|
+
rounds += 1
|
|
129
|
+
results = execute_pipeline(raw_connection_for(key), pending)
|
|
130
|
+
index = results.index { |result| result.is_a?(Exception) }
|
|
131
|
+
|
|
132
|
+
if index
|
|
133
|
+
failed += 1
|
|
134
|
+
resolutions[pending.delete_at(index)] = [:reject, results[index]]
|
|
135
|
+
elsif results.any?(ABORTED)
|
|
136
|
+
raise Error, "pipeline aborted without reporting an error"
|
|
137
|
+
else
|
|
138
|
+
pending.zip(results) { |item, result| resolutions[item] = [:fulfill, result] }
|
|
139
|
+
pending = []
|
|
140
|
+
end
|
|
141
|
+
end
|
|
142
|
+
ensure
|
|
143
|
+
event[:rounds] = rounds
|
|
144
|
+
event[:failed] = failed
|
|
145
|
+
end
|
|
146
|
+
|
|
147
|
+
def execute_pipeline(conn, items)
|
|
148
|
+
conn.enter_pipeline_mode
|
|
149
|
+
items.each { |item| conn.send_query_params(item.sql, item.binds) }
|
|
150
|
+
conn.pipeline_sync
|
|
151
|
+
|
|
152
|
+
results = items.map { |item| convert(read_result(conn), item) }
|
|
153
|
+
read_sync(conn)
|
|
154
|
+
conn.exit_pipeline_mode
|
|
155
|
+
results
|
|
156
|
+
end
|
|
157
|
+
|
|
158
|
+
# In pipeline mode each statement's results are terminated by a nil.
|
|
159
|
+
def read_result(conn)
|
|
160
|
+
last = nil
|
|
161
|
+
while (result = conn.get_result)
|
|
162
|
+
last = result
|
|
163
|
+
end
|
|
164
|
+
last
|
|
165
|
+
end
|
|
166
|
+
|
|
167
|
+
def convert(result, item)
|
|
168
|
+
case result&.result_status
|
|
169
|
+
when PG::PGRES_TUPLES_OK, PG::PGRES_COMMAND_OK
|
|
170
|
+
value = item.operation == :insert ? ActiveRecord::Result.new(result.fields, result.values) : result.cmd_tuples
|
|
171
|
+
result.clear
|
|
172
|
+
value
|
|
173
|
+
when PG::PGRES_PIPELINE_ABORTED
|
|
174
|
+
ABORTED
|
|
175
|
+
when PG::PGRES_FATAL_ERROR
|
|
176
|
+
error_for(result)
|
|
177
|
+
else
|
|
178
|
+
raise Error, "unexpected result status #{result&.result_status.inspect} for #{item.sql}"
|
|
179
|
+
end
|
|
180
|
+
end
|
|
181
|
+
|
|
182
|
+
# PG::Result#check raises the specific PG::Error subclass (UniqueViolation,
|
|
183
|
+
# NotNullViolation...) with the result attached, which is what ActiveRecord
|
|
184
|
+
# needs to translate it into RecordNotUnique, NotNullViolation, etc.
|
|
185
|
+
def error_for(result)
|
|
186
|
+
result.check
|
|
187
|
+
Error.new("statement failed without an error message")
|
|
188
|
+
rescue PG::Error => e
|
|
189
|
+
e
|
|
190
|
+
end
|
|
191
|
+
|
|
192
|
+
# The implicit transaction commits at the sync point. An error there (e.g. a
|
|
193
|
+
# deferred constraint) can't be attributed to a statement, so it is raised
|
|
194
|
+
# and rejects the whole group.
|
|
195
|
+
def read_sync(conn)
|
|
196
|
+
result = conn.get_result
|
|
197
|
+
return if result&.result_status == PG::PGRES_PIPELINE_SYNC
|
|
198
|
+
|
|
199
|
+
result&.check
|
|
200
|
+
raise Error, "pipeline sync was not acknowledged"
|
|
201
|
+
end
|
|
202
|
+
|
|
203
|
+
# Each group gets its own ActiveRecord adapter so the session is configured
|
|
204
|
+
# exactly like the application's (search_path, timezone, variables...).
|
|
205
|
+
def raw_connection_for(key)
|
|
206
|
+
adapter = @adapters.compute_if_absent(key) do
|
|
207
|
+
ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.new(key.first.configuration_hash)
|
|
208
|
+
end
|
|
209
|
+
adapter.raw_connection
|
|
210
|
+
end
|
|
211
|
+
|
|
212
|
+
def discard_adapter(key)
|
|
213
|
+
adapter = @adapters.delete(key)
|
|
214
|
+
adapter&.disconnect!
|
|
215
|
+
rescue StandardError
|
|
216
|
+
nil
|
|
217
|
+
end
|
|
218
|
+
|
|
219
|
+
def disconnect_all
|
|
220
|
+
@adapters.each_key { |key| discard_adapter(key) }
|
|
221
|
+
end
|
|
222
|
+
|
|
223
|
+
def reject_leftovers(error)
|
|
224
|
+
while (payload = @queue.pop(timeout: 0))
|
|
225
|
+
payload.reject(error)
|
|
226
|
+
end
|
|
227
|
+
end
|
|
228
|
+
|
|
229
|
+
def monotonic_now
|
|
230
|
+
Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
231
|
+
end
|
|
232
|
+
end
|
|
233
|
+
end
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "concurrent"
|
|
4
|
+
require "active_record"
|
|
5
|
+
require "pg"
|
|
6
|
+
|
|
7
|
+
require_relative "active_pipeline/version"
|
|
8
|
+
require_relative "active_pipeline/payload"
|
|
9
|
+
require_relative "active_pipeline/worker"
|
|
10
|
+
require_relative "active_pipeline/adapter_extension"
|
|
11
|
+
|
|
12
|
+
module ActivePipeline
|
|
13
|
+
class Error < StandardError; end
|
|
14
|
+
|
|
15
|
+
# Raised in the caller thread when +wait_timeout+ elapses before the worker
|
|
16
|
+
# resolved the statement. The statement may still be committed afterwards.
|
|
17
|
+
class TimeoutError < Error; end
|
|
18
|
+
|
|
19
|
+
# ActiveSupport::Notifications event emitted once per (database, table) group
|
|
20
|
+
# flushed by the worker. Payload: :table, :size, :rounds, :failed.
|
|
21
|
+
BATCH_EVENT = "batch.active_pipeline"
|
|
22
|
+
|
|
23
|
+
@mutex = Mutex.new
|
|
24
|
+
|
|
25
|
+
class << self
|
|
26
|
+
# Master switch. When false every write goes straight through ActiveRecord.
|
|
27
|
+
attr_accessor :enabled
|
|
28
|
+
|
|
29
|
+
# How long the worker keeps collecting statements after the first one arrives.
|
|
30
|
+
attr_accessor :batch_window_ms
|
|
31
|
+
|
|
32
|
+
# Upper bound of statements flushed in a single pipeline.
|
|
33
|
+
attr_accessor :max_batch_size
|
|
34
|
+
|
|
35
|
+
# Seconds a caller waits for its statement to be resolved (nil waits forever).
|
|
36
|
+
attr_accessor :wait_timeout
|
|
37
|
+
|
|
38
|
+
def configure
|
|
39
|
+
yield self
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def start!
|
|
43
|
+
@mutex.synchronize do
|
|
44
|
+
@worker = Worker.new.tap(&:start!) unless @worker&.running?
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def stop!
|
|
49
|
+
@mutex.synchronize do
|
|
50
|
+
@worker&.stop!
|
|
51
|
+
@worker = nil
|
|
52
|
+
end
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
def running?
|
|
56
|
+
!!@worker&.running?
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
# Runs the block with the pipeline disabled for the current thread/fiber.
|
|
60
|
+
def bypass
|
|
61
|
+
previous = Thread.current[:active_pipeline_bypass]
|
|
62
|
+
Thread.current[:active_pipeline_bypass] = true
|
|
63
|
+
yield
|
|
64
|
+
ensure
|
|
65
|
+
Thread.current[:active_pipeline_bypass] = previous
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
def bypassed?
|
|
69
|
+
!!Thread.current[:active_pipeline_bypass]
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
# Hands a payload to the running worker. Returns false when nothing is
|
|
73
|
+
# running so the caller can fall back to a regular, synchronous write.
|
|
74
|
+
def enqueue(payload) # :nodoc:
|
|
75
|
+
!!@worker&.enqueue(payload)
|
|
76
|
+
end
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
self.enabled = true
|
|
80
|
+
self.batch_window_ms = 50
|
|
81
|
+
self.max_batch_size = 500
|
|
82
|
+
self.wait_timeout = nil
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
ActiveSupport.on_load(:active_record_postgresqladapter) do
|
|
86
|
+
prepend ActivePipeline::AdapterExtension
|
|
87
|
+
end
|
metadata
ADDED
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: active_pipeline
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.1.0
|
|
5
|
+
platform: ruby
|
|
6
|
+
authors:
|
|
7
|
+
- Carlos (Thrashattack) Cunha
|
|
8
|
+
bindir: bin
|
|
9
|
+
cert_chain: []
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
|
+
dependencies:
|
|
12
|
+
- !ruby/object:Gem::Dependency
|
|
13
|
+
name: activerecord
|
|
14
|
+
requirement: !ruby/object:Gem::Requirement
|
|
15
|
+
requirements:
|
|
16
|
+
- - ">="
|
|
17
|
+
- !ruby/object:Gem::Version
|
|
18
|
+
version: '7.1'
|
|
19
|
+
type: :runtime
|
|
20
|
+
prerelease: false
|
|
21
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
22
|
+
requirements:
|
|
23
|
+
- - ">="
|
|
24
|
+
- !ruby/object:Gem::Version
|
|
25
|
+
version: '7.1'
|
|
26
|
+
- !ruby/object:Gem::Dependency
|
|
27
|
+
name: concurrent-ruby
|
|
28
|
+
requirement: !ruby/object:Gem::Requirement
|
|
29
|
+
requirements:
|
|
30
|
+
- - "~>"
|
|
31
|
+
- !ruby/object:Gem::Version
|
|
32
|
+
version: '1.1'
|
|
33
|
+
type: :runtime
|
|
34
|
+
prerelease: false
|
|
35
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
36
|
+
requirements:
|
|
37
|
+
- - "~>"
|
|
38
|
+
- !ruby/object:Gem::Version
|
|
39
|
+
version: '1.1'
|
|
40
|
+
- !ruby/object:Gem::Dependency
|
|
41
|
+
name: pg
|
|
42
|
+
requirement: !ruby/object:Gem::Requirement
|
|
43
|
+
requirements:
|
|
44
|
+
- - ">="
|
|
45
|
+
- !ruby/object:Gem::Version
|
|
46
|
+
version: '1.3'
|
|
47
|
+
type: :runtime
|
|
48
|
+
prerelease: false
|
|
49
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
50
|
+
requirements:
|
|
51
|
+
- - ">="
|
|
52
|
+
- !ruby/object:Gem::Version
|
|
53
|
+
version: '1.3'
|
|
54
|
+
description: Implements Go-channel-like write batching for Rails and Sidekiq using
|
|
55
|
+
PostgreSQL pipeline mode, with per-statement error isolation.
|
|
56
|
+
email:
|
|
57
|
+
- henrique2092@gmail.com
|
|
58
|
+
executables: []
|
|
59
|
+
extensions: []
|
|
60
|
+
extra_rdoc_files: []
|
|
61
|
+
files:
|
|
62
|
+
- CHANGELOG.md
|
|
63
|
+
- LICENSE.txt
|
|
64
|
+
- README.md
|
|
65
|
+
- lib/active_pipeline.rb
|
|
66
|
+
- lib/active_pipeline/adapter_extension.rb
|
|
67
|
+
- lib/active_pipeline/payload.rb
|
|
68
|
+
- lib/active_pipeline/version.rb
|
|
69
|
+
- lib/active_pipeline/worker.rb
|
|
70
|
+
homepage: https://github.com/thrashattack/active_pipeline
|
|
71
|
+
licenses:
|
|
72
|
+
- MIT
|
|
73
|
+
metadata:
|
|
74
|
+
homepage_uri: https://github.com/thrashattack/active_pipeline
|
|
75
|
+
source_code_uri: https://github.com/thrashattack/active_pipeline
|
|
76
|
+
changelog_uri: https://github.com/thrashattack/active_pipeline/blob/main/CHANGELOG.md
|
|
77
|
+
bug_tracker_uri: https://github.com/thrashattack/active_pipeline/issues
|
|
78
|
+
rubygems_mfa_required: 'true'
|
|
79
|
+
rdoc_options: []
|
|
80
|
+
require_paths:
|
|
81
|
+
- lib
|
|
82
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
83
|
+
requirements:
|
|
84
|
+
- - ">="
|
|
85
|
+
- !ruby/object:Gem::Version
|
|
86
|
+
version: '3.2'
|
|
87
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
88
|
+
requirements:
|
|
89
|
+
- - ">="
|
|
90
|
+
- !ruby/object:Gem::Version
|
|
91
|
+
version: '0'
|
|
92
|
+
requirements: []
|
|
93
|
+
rubygems_version: 3.6.9
|
|
94
|
+
specification_version: 4
|
|
95
|
+
summary: Transparent PostgreSQL pipeline batching for ActiveRecord.
|
|
96
|
+
test_files: []
|