pg_pipeline 0.2.2 → 0.2.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +49 -0
- data/DESIGN.md +24 -4
- data/README.md +31 -1
- data/lib/pg_pipeline/client.rb +27 -5
- data/lib/pg_pipeline/connection_driver.rb +57 -37
- data/lib/pg_pipeline/pool.rb +100 -20
- data/lib/pg_pipeline/prepared_statement.rb +47 -0
- data/lib/pg_pipeline/request.rb +55 -3
- data/lib/pg_pipeline/session_guard.rb +10 -3
- data/lib/pg_pipeline/version.rb +1 -1
- data/lib/pg_pipeline.rb +1 -0
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 7135257ec52d199f4a7e0d3666909eebd8effa277f7efe80d27ca8ff00717f93
|
|
4
|
+
data.tar.gz: 8f3b976cc4b9f8400439a1504bbad9a019865a88a7f41e6556907ba649587c80
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 90422fd9a948509c2e53b9058d052a822e76f303a736a39488a0870b6fa903d6c1b2b8248e00649269a63ed606a9a8d04b1c2c081cf0f2b89e81000af8a36e54
|
|
7
|
+
data.tar.gz: fa76f361560fa078903685d69de81600b8337509b8f37b89cd93fdfdc71ebc3953180edf8487b4aa24d79d521237ef98bc727cde48ca959acb41c0d589749999
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,54 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.2.3] - 2026-07-30
|
|
4
|
+
|
|
5
|
+
Performance-focused release based on CPU profiles from the live pipeline
|
|
6
|
+
workload. It removes redundant libpq polling, adds an explicit prepared-
|
|
7
|
+
statement hot path, and trims several per-query control-plane costs without
|
|
8
|
+
changing the failure model.
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- `Client#prepare(name, sql, param_types = nil)` prepares immutable SQL on every
|
|
13
|
+
currently live pipeline connection and returns a `PreparedStatement` handle.
|
|
14
|
+
- `PreparedStatement#query(params = [])` executes through
|
|
15
|
+
`send_query_prepared`, avoiding repeated Parse/Describe work for hot SQL.
|
|
16
|
+
- Replacement pipeline connections replay the registered prepared-statement
|
|
17
|
+
catalog before becoming available, including registrations that race with a
|
|
18
|
+
reconnect.
|
|
19
|
+
- Prepared-statement unit and live integration coverage, including execution
|
|
20
|
+
across multiple drivers and automatic re-prepare after reconnect.
|
|
21
|
+
|
|
22
|
+
### Performance
|
|
23
|
+
|
|
24
|
+
- The connection owner drains results only after `consume_input` actually read
|
|
25
|
+
a readable socket event. Removed unconditional pre/post-dispatch drain passes
|
|
26
|
+
that repeatedly called `PQisBusy` without new input.
|
|
27
|
+
- A blocked `PQflush` is retried by the writer watcher instead of again on every
|
|
28
|
+
unrelated owner event. Newly queued output remains buffered in libpq until the
|
|
29
|
+
socket becomes writable.
|
|
30
|
+
- Driver selection is now one circular least-load pass: each available driver's
|
|
31
|
+
`load` is read once and equal-load ties rotate from the selected slot.
|
|
32
|
+
- The normalized SQL guard mode is reused on the query hot path instead of
|
|
33
|
+
coercing and validating it for every request.
|
|
34
|
+
- Already-frozen SQL strings and frozen string bind values are reused rather
|
|
35
|
+
than duplicated; mutable caller input is still snapshotted before submission.
|
|
36
|
+
- Prepared-only metadata lives on internal request subclasses, so ordinary
|
|
37
|
+
query requests do not grow extra instance variables for the new API.
|
|
38
|
+
- The throughput and A/B harnesses explicitly clear every result and support
|
|
39
|
+
`PREPARED=1`, preparing both pipeline and baseline clients for a fair hot-path
|
|
40
|
+
comparison.
|
|
41
|
+
|
|
42
|
+
### Semantics
|
|
43
|
+
|
|
44
|
+
- Prepared statements are explicit client-scoped handles, not an automatic
|
|
45
|
+
unbounded SQL cache. The SQL guard runs once during registration.
|
|
46
|
+
- Failed registration is removed from the reconnect catalog. Successfully
|
|
47
|
+
prepared but unreachable internal names may remain on already-prepared
|
|
48
|
+
physical connections until those connections close.
|
|
49
|
+
- Multiplexed prepared handles live for the lifetime of the client; this release
|
|
50
|
+
intentionally does not add a concurrent `DEALLOCATE` API.
|
|
51
|
+
|
|
3
52
|
## [0.2.2] - 2026-07-30
|
|
4
53
|
|
|
5
54
|
Lifecycle-hardening release. Fixes five pool/task-ownership defects surfaced by
|
data/DESIGN.md
CHANGED
|
@@ -51,10 +51,13 @@ sync_get_result
|
|
|
51
51
|
`is_busy` is only a readiness predicate: false means the next
|
|
52
52
|
`sync_get_result` will not block. It is not a request boundary.
|
|
53
53
|
|
|
54
|
-
The owner
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
54
|
+
The owner calls `consume_input` on a socket-readable event and then drains until
|
|
55
|
+
`is_busy` becomes true or the in-flight FIFO is empty. A drain pass exhausts all
|
|
56
|
+
currently buffered results, including locally represented pipeline-aborted and
|
|
57
|
+
pipeline-sync statuses. Request, writable and bookkeeping events therefore do
|
|
58
|
+
not poll `is_busy` again: without a new `consume_input` they cannot reveal new
|
|
59
|
+
server input, and the redundant Ruby-to-C calls were the largest control-plane
|
|
60
|
+
CPU hot spot in profiling.
|
|
58
61
|
|
|
59
62
|
## 4. Protocol/FIFO model
|
|
60
63
|
|
|
@@ -112,6 +115,21 @@ Anything stateful goes through an exclusive pinned connection:
|
|
|
112
115
|
- `Client#session` — no implicit BEGIN;
|
|
113
116
|
- `Client#transaction` — BEGIN/COMMIT/ROLLBACK.
|
|
114
117
|
|
|
118
|
+
The one deliberate exception is `Client#prepare`, which is not an arbitrary
|
|
119
|
+
session mutation exposed to callers. It registers immutable SQL in a
|
|
120
|
+
client-owned catalog and prepares the same generated physical name on every
|
|
121
|
+
pipeline connection. The returned `PreparedStatement` handle can then use
|
|
122
|
+
`send_query_prepared` through the normal FIFO protocol. A replacement connection
|
|
123
|
+
replays the full catalog before it becomes available, so routing never selects a
|
|
124
|
+
driver that lacks a successfully registered handle.
|
|
125
|
+
|
|
126
|
+
Registration is explicit: there is no unbounded automatic SQL cache. A failed
|
|
127
|
+
registration is removed from the reconnect catalog; statements that had already
|
|
128
|
+
been prepared on another connection may remain there under an unreachable
|
|
129
|
+
generated name until that connection is replaced or closed. Version 0.2.3 does
|
|
130
|
+
not expose `DEALLOCATE` for multiplexed handles. Server-side plan invalidation is
|
|
131
|
+
reported as the ordinary request-local `QueryError`.
|
|
132
|
+
|
|
115
133
|
Pinned connections are opened lazily up to `pinned_size`; a query-only process
|
|
116
134
|
does not reserve an otherwise idle second pool at startup. Before a pinned
|
|
117
135
|
physical connection is reused, the pool makes sure it is outside an explicit
|
|
@@ -250,6 +268,8 @@ chunked-rows, non-blocking cancel), never a floor.
|
|
|
250
268
|
(`cancel_pinned_on_abort: true`); it does not force-close the socket if the
|
|
251
269
|
backend ignores cancel.
|
|
252
270
|
- No streaming/single-row/chunked-row result API.
|
|
271
|
+
- No multiplexed prepared-statement deallocation API; registered handles live
|
|
272
|
+
for the lifetime of the client.
|
|
253
273
|
- No cross-thread/reactor sharing; `Client` rejects use from a different thread or Fiber scheduler.
|
|
254
274
|
- `SessionGuard` is policy/ergonomics, not a security boundary. `:strict` adds a
|
|
255
275
|
precise denylist (`nextval`/`setval`/`pg_export_snapshot`, …) without the old
|
data/README.md
CHANGED
|
@@ -62,6 +62,35 @@ end
|
|
|
62
62
|
`Client.open` manages the pool lifecycle. `db.query` is fiber-safe and multiplexed —
|
|
63
63
|
every fiber yields while waiting, the event loop stays unblocked.
|
|
64
64
|
|
|
65
|
+
### Multiplexed prepared statements
|
|
66
|
+
|
|
67
|
+
For repeated SQL, prepare it once on every pipeline connection and execute the
|
|
68
|
+
returned immutable handle:
|
|
69
|
+
|
|
70
|
+
```ruby
|
|
71
|
+
by_id = db.prepare(
|
|
72
|
+
"user_by_id",
|
|
73
|
+
"SELECT id, email, name FROM users WHERE id = $1",
|
|
74
|
+
[23] # optional PostgreSQL parameter OIDs
|
|
75
|
+
)
|
|
76
|
+
|
|
77
|
+
result = by_id.query([42])
|
|
78
|
+
begin
|
|
79
|
+
p result.first
|
|
80
|
+
ensure
|
|
81
|
+
result.clear
|
|
82
|
+
end
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
`Client#prepare` returns only after the statement is ready on every currently
|
|
86
|
+
live pipeline connection. Replacement connections automatically prepare the
|
|
87
|
+
registered catalog before they begin accepting requests. The logical name is
|
|
88
|
+
client-local; the gem uses private generated server names on its owned connections.
|
|
89
|
+
|
|
90
|
+
Prepared handles are intentionally explicit rather than an unbounded automatic
|
|
91
|
+
SQL cache. The SQL guard runs once at preparation time, while each execution
|
|
92
|
+
still snapshots its bind values before submission.
|
|
93
|
+
|
|
65
94
|
### Transactions
|
|
66
95
|
|
|
67
96
|
```ruby
|
|
@@ -84,7 +113,7 @@ db.transaction do |tx|
|
|
|
84
113
|
end
|
|
85
114
|
```
|
|
86
115
|
|
|
87
|
-
### Session (DDL, LISTEN, SET, prepared statements)
|
|
116
|
+
### Session (DDL, LISTEN, SET, connection-local prepared statements)
|
|
88
117
|
|
|
89
118
|
```ruby
|
|
90
119
|
db.session do |s|
|
|
@@ -119,6 +148,7 @@ db.stats
|
|
|
119
148
|
# => {
|
|
120
149
|
# pipeline: { size: 4, live: 4, drivers: [{load: 3, in_flight: 2, ...}, ...] },
|
|
121
150
|
# pinned: { size: 2, active: 1, free: 1 },
|
|
151
|
+
# prepared_statements: 1,
|
|
122
152
|
# reconnects: 0
|
|
123
153
|
# }
|
|
124
154
|
```
|
data/lib/pg_pipeline/client.rb
CHANGED
|
@@ -5,6 +5,7 @@ require "async"
|
|
|
5
5
|
require_relative "errors"
|
|
6
6
|
require_relative "pool"
|
|
7
7
|
require_relative "request"
|
|
8
|
+
require_relative "prepared_statement"
|
|
8
9
|
require_relative "session_guard"
|
|
9
10
|
require_relative "session"
|
|
10
11
|
require_relative "transaction"
|
|
@@ -27,6 +28,7 @@ module PgPipeline
|
|
|
27
28
|
|
|
28
29
|
def start(parent: Async::Task.current) = ClientOps.start(self, parent)
|
|
29
30
|
def query(sql, params = []) = ClientOps.query(self, sql, params)
|
|
31
|
+
def prepare(name, sql, param_types = nil) = ClientOps.prepare(self, name, sql, param_types)
|
|
30
32
|
def stats = ClientOps.stats(self)
|
|
31
33
|
|
|
32
34
|
def session(&block)
|
|
@@ -71,11 +73,31 @@ module PgPipeline
|
|
|
71
73
|
|
|
72
74
|
def query(client, sql, params)
|
|
73
75
|
ensure_started!(client)
|
|
74
|
-
sql = sql
|
|
75
|
-
SessionGuard.
|
|
76
|
+
sql = RequestOps.snapshot_sql(sql)
|
|
77
|
+
SessionGuard.assert_multiplexable_normalized!(sql, mode: client.guard)
|
|
76
78
|
|
|
77
|
-
|
|
79
|
+
wait_for_request do
|
|
80
|
+
submit_with_failover(client) { Request.new(sql: sql, params: params) }
|
|
81
|
+
end
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
def prepare(client, name, sql, param_types)
|
|
85
|
+
ensure_started!(client)
|
|
86
|
+
sql = RequestOps.snapshot_sql(sql)
|
|
87
|
+
SessionGuard.assert_multiplexable_normalized!(sql, mode: client.guard)
|
|
88
|
+
pool(client).__send__(:prepare_statement, client, name, sql, param_types)
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def query_prepared(client, statement, params)
|
|
92
|
+
ensure_started!(client)
|
|
93
|
+
|
|
94
|
+
wait_for_request do
|
|
95
|
+
submit_with_failover(client) { Request.prepared_query(statement, params: params) }
|
|
96
|
+
end
|
|
97
|
+
end
|
|
78
98
|
|
|
99
|
+
def wait_for_request
|
|
100
|
+
request = yield
|
|
79
101
|
begin
|
|
80
102
|
request.wait
|
|
81
103
|
ensure
|
|
@@ -85,13 +107,13 @@ module PgPipeline
|
|
|
85
107
|
|
|
86
108
|
# NotDispatchedError is safe to retry on a fresh Request by contract.
|
|
87
109
|
# ShutdownError is retried only while the current Request is still pre-dispatch.
|
|
88
|
-
def submit_with_failover(client
|
|
110
|
+
def submit_with_failover(client)
|
|
89
111
|
attempts = 0
|
|
90
112
|
limit = [pool(client).pipeline_size, 1].max
|
|
91
113
|
last_error = nil
|
|
92
114
|
|
|
93
115
|
while attempts < limit
|
|
94
|
-
request =
|
|
116
|
+
request = yield
|
|
95
117
|
begin
|
|
96
118
|
pool(client).__send__(:pipeline_driver).submit(request)
|
|
97
119
|
return request
|
|
@@ -196,44 +196,46 @@ module PgPipeline
|
|
|
196
196
|
end
|
|
197
197
|
|
|
198
198
|
def owner_loop(d)
|
|
199
|
-
while d.running
|
|
200
|
-
event = d.events.dequeue
|
|
201
|
-
|
|
202
|
-
case event
|
|
203
|
-
when :requests
|
|
204
|
-
d.request_event_pending = false
|
|
205
|
-
when :submission_finished
|
|
206
|
-
nil
|
|
207
|
-
when :readable
|
|
208
|
-
begin
|
|
209
|
-
read_available(d)
|
|
210
|
-
ensure
|
|
211
|
-
d.reader_rearm.enqueue(:rearm) if d.running
|
|
212
|
-
end
|
|
213
|
-
when :writable
|
|
214
|
-
d.writer_armed = false
|
|
215
|
-
flush_output(d)
|
|
216
|
-
when :close
|
|
217
|
-
d.draining = true
|
|
218
|
-
when Array
|
|
219
|
-
tag, payload = event
|
|
220
|
-
if tag == :abort
|
|
221
|
-
fatal_close(d, payload)
|
|
222
|
-
break
|
|
223
|
-
end
|
|
224
|
-
end
|
|
225
|
-
|
|
226
|
-
drain_results(d)
|
|
227
|
-
pump_requests(d)
|
|
228
|
-
drain_results(d)
|
|
229
|
-
finish_graceful_close(d) if d.draining && drained?(d)
|
|
230
|
-
end
|
|
199
|
+
process_event(d, d.events.dequeue) while d.running
|
|
231
200
|
rescue StandardError => e
|
|
232
201
|
fatal_close(d, ConnectionLostError.new("driver crashed: #{e.class}: #{e.message}"))
|
|
233
202
|
ensure
|
|
234
203
|
fatal_close(d, ShutdownError.new("driver owner stopped before shutdown completed")) if d.running
|
|
235
204
|
end
|
|
236
205
|
|
|
206
|
+
def process_event(d, event)
|
|
207
|
+
input_changed = false
|
|
208
|
+
|
|
209
|
+
case event
|
|
210
|
+
when :requests
|
|
211
|
+
d.request_event_pending = false
|
|
212
|
+
when :submission_finished
|
|
213
|
+
nil
|
|
214
|
+
when :readable
|
|
215
|
+
begin
|
|
216
|
+
read_available(d)
|
|
217
|
+
input_changed = true
|
|
218
|
+
ensure
|
|
219
|
+
d.reader_rearm.enqueue(:rearm) if d.running
|
|
220
|
+
end
|
|
221
|
+
when :writable
|
|
222
|
+
d.writer_armed = false
|
|
223
|
+
flush_output(d)
|
|
224
|
+
when :close
|
|
225
|
+
d.draining = true
|
|
226
|
+
when Array
|
|
227
|
+
tag, payload = event
|
|
228
|
+
if tag == :abort
|
|
229
|
+
fatal_close(d, payload)
|
|
230
|
+
return
|
|
231
|
+
end
|
|
232
|
+
end
|
|
233
|
+
|
|
234
|
+
drain_results(d) if input_changed
|
|
235
|
+
pump_requests(d) if d.running
|
|
236
|
+
finish_graceful_close(d) if d.running && d.draining && drained?(d)
|
|
237
|
+
end
|
|
238
|
+
|
|
237
239
|
def notify_requests(d)
|
|
238
240
|
return if d.request_event_pending
|
|
239
241
|
|
|
@@ -261,12 +263,12 @@ module PgPipeline
|
|
|
261
263
|
dispatched = true
|
|
262
264
|
end
|
|
263
265
|
|
|
264
|
-
flush_output(d) if dispatched
|
|
266
|
+
flush_output(d) if dispatched && !d.needs_flush
|
|
265
267
|
end
|
|
266
268
|
|
|
267
269
|
def send_unit(d, request)
|
|
268
270
|
begin
|
|
269
|
-
d.conn
|
|
271
|
+
send_command(d.conn, request)
|
|
270
272
|
rescue PG::UnableToSend => e
|
|
271
273
|
d.dispatching = nil
|
|
272
274
|
request.reject!(
|
|
@@ -281,9 +283,11 @@ module PgPipeline
|
|
|
281
283
|
NotDispatchedError.new("query was rejected before libpq accepted it: #{e.class}: #{e.message}")
|
|
282
284
|
)
|
|
283
285
|
raise ConnectionLostError, "dispatch failed: #{e.class}: #{e.message}"
|
|
286
|
+
rescue ProtocolError
|
|
287
|
+
raise
|
|
284
288
|
rescue StandardError => e
|
|
285
|
-
# ruby-pg prepares/encodes query parameters before calling
|
|
286
|
-
#
|
|
289
|
+
# ruby-pg prepares/encodes query parameters before calling PQsend*. A
|
|
290
|
+
# Ruby-side encoder/coercion exception therefore belongs only to this
|
|
287
291
|
# request and must not poison unrelated work already in the pipeline.
|
|
288
292
|
request.reject!(e)
|
|
289
293
|
return false
|
|
@@ -295,6 +299,23 @@ module PgPipeline
|
|
|
295
299
|
raise ConnectionLostError, "dispatch Sync failed: #{e.class}: #{e.message}"
|
|
296
300
|
end
|
|
297
301
|
|
|
302
|
+
def send_command(conn, request)
|
|
303
|
+
case request.operation
|
|
304
|
+
when :query
|
|
305
|
+
conn.send_query_params(request.sql, request.params)
|
|
306
|
+
when :prepare
|
|
307
|
+
if request.param_types.nil?
|
|
308
|
+
conn.send_prepare(request.statement_name, request.sql)
|
|
309
|
+
else
|
|
310
|
+
conn.send_prepare(request.statement_name, request.sql, request.param_types)
|
|
311
|
+
end
|
|
312
|
+
when :prepared_query
|
|
313
|
+
conn.send_query_prepared(request.statement_name, request.params)
|
|
314
|
+
else
|
|
315
|
+
raise ProtocolError, "unsupported request operation #{request.operation.inspect}"
|
|
316
|
+
end
|
|
317
|
+
end
|
|
318
|
+
|
|
298
319
|
def reusable_after_send_rejection?(d)
|
|
299
320
|
!d.conn.finished? &&
|
|
300
321
|
d.conn.status == PG::CONNECTION_OK &&
|
|
@@ -325,7 +346,6 @@ module PgPipeline
|
|
|
325
346
|
|
|
326
347
|
def read_available(d)
|
|
327
348
|
d.conn.consume_input
|
|
328
|
-
drain_results(d)
|
|
329
349
|
rescue PG::Error => e
|
|
330
350
|
raise ConnectionLostError, "read failed: #{e.class}: #{e.message}"
|
|
331
351
|
end
|
data/lib/pg_pipeline/pool.rb
CHANGED
|
@@ -7,6 +7,7 @@ require "async/notification"
|
|
|
7
7
|
|
|
8
8
|
require_relative "errors"
|
|
9
9
|
require_relative "connection_driver"
|
|
10
|
+
require_relative "prepared_statement"
|
|
10
11
|
require_relative "server_caps"
|
|
11
12
|
|
|
12
13
|
module PgPipeline
|
|
@@ -58,6 +59,10 @@ module PgPipeline
|
|
|
58
59
|
@supervisor = nil
|
|
59
60
|
@task_parent = nil
|
|
60
61
|
|
|
62
|
+
@prepared_statements = {}
|
|
63
|
+
@prepared_generation = 0
|
|
64
|
+
@statement_sequence = 0
|
|
65
|
+
|
|
61
66
|
@pinned_free = []
|
|
62
67
|
@pinned_in_use = {}
|
|
63
68
|
@pinned_gate = nil
|
|
@@ -107,6 +112,61 @@ module PgPipeline
|
|
|
107
112
|
driver
|
|
108
113
|
end
|
|
109
114
|
|
|
115
|
+
def prepare_statement(client, name, sql, param_types)
|
|
116
|
+
ensure_available!
|
|
117
|
+
logical_name = PreparedStatementOps.snapshot_name(name)
|
|
118
|
+
if @prepared_statements.key?(logical_name)
|
|
119
|
+
raise Error, "prepared statement #{logical_name.inspect} already exists"
|
|
120
|
+
end
|
|
121
|
+
|
|
122
|
+
@statement_sequence += 1
|
|
123
|
+
statement = PreparedStatement.new(
|
|
124
|
+
client: client,
|
|
125
|
+
name: logical_name,
|
|
126
|
+
physical_name: "pgp_#{@statement_sequence.to_s(36)}",
|
|
127
|
+
sql: sql,
|
|
128
|
+
param_types: param_types
|
|
129
|
+
)
|
|
130
|
+
|
|
131
|
+
@prepared_statements[logical_name] = statement
|
|
132
|
+
@prepared_generation += 1
|
|
133
|
+
requests = []
|
|
134
|
+
|
|
135
|
+
begin
|
|
136
|
+
drivers = @drivers.select(&:available?)
|
|
137
|
+
if drivers.empty?
|
|
138
|
+
raise NotDispatchedError, "no live pipeline connections; statement was not prepared"
|
|
139
|
+
end
|
|
140
|
+
|
|
141
|
+
requests = drivers.map do |driver|
|
|
142
|
+
Request.prepare(statement).tap { |request| driver.submit(request) }
|
|
143
|
+
end
|
|
144
|
+
|
|
145
|
+
first_error = nil
|
|
146
|
+
requests.each do |request|
|
|
147
|
+
begin
|
|
148
|
+
RequestOps.clear_result(request.wait)
|
|
149
|
+
rescue StandardError => e
|
|
150
|
+
if first_error
|
|
151
|
+
e.clear_result! if e.respond_to?(:clear_result!)
|
|
152
|
+
else
|
|
153
|
+
first_error = e
|
|
154
|
+
end
|
|
155
|
+
end
|
|
156
|
+
end
|
|
157
|
+
raise first_error if first_error
|
|
158
|
+
|
|
159
|
+
statement
|
|
160
|
+
rescue Exception
|
|
161
|
+
if @prepared_statements.delete(logical_name)
|
|
162
|
+
@prepared_generation += 1
|
|
163
|
+
end
|
|
164
|
+
raise
|
|
165
|
+
ensure
|
|
166
|
+
requests.each { |request| request.cancel! unless request.settled? }
|
|
167
|
+
end
|
|
168
|
+
end
|
|
169
|
+
|
|
110
170
|
def with_pinned
|
|
111
171
|
ensure_available!
|
|
112
172
|
raise @pinned_error if @pinned_error
|
|
@@ -154,6 +214,7 @@ module PgPipeline
|
|
|
154
214
|
free: @pinned_free.size,
|
|
155
215
|
in_use: @pinned_in_use.size
|
|
156
216
|
},
|
|
217
|
+
prepared_statements: (@prepared_statements || {}).size,
|
|
157
218
|
reconnects: @reconnects,
|
|
158
219
|
health_failures: @health_failures,
|
|
159
220
|
supervisor_error: @supervisor_error&.message,
|
|
@@ -352,6 +413,7 @@ module PgPipeline
|
|
|
352
413
|
|
|
353
414
|
def start_pipeline_driver(parent)
|
|
354
415
|
conn = PoolOps.new_connection(@connection_args)
|
|
416
|
+
prepare_registered_statements(conn)
|
|
355
417
|
driver = ConnectionDriver.new(conn, max_pending: @max_pending, max_in_flight: @max_in_flight)
|
|
356
418
|
driver.start(parent: parent)
|
|
357
419
|
rescue Exception
|
|
@@ -359,6 +421,29 @@ module PgPipeline
|
|
|
359
421
|
raise
|
|
360
422
|
end
|
|
361
423
|
|
|
424
|
+
def prepare_registered_statements(conn)
|
|
425
|
+
prepared = {}
|
|
426
|
+
|
|
427
|
+
loop do
|
|
428
|
+
generation = @prepared_generation || 0
|
|
429
|
+
statements = (@prepared_statements || {}).values.dup
|
|
430
|
+
|
|
431
|
+
statements.each do |statement|
|
|
432
|
+
next if prepared.key?(statement.physical_name)
|
|
433
|
+
|
|
434
|
+
result = if statement.param_types.nil?
|
|
435
|
+
conn.prepare(statement.physical_name, statement.sql)
|
|
436
|
+
else
|
|
437
|
+
conn.prepare(statement.physical_name, statement.sql, statement.param_types)
|
|
438
|
+
end
|
|
439
|
+
RequestOps.clear_result(result)
|
|
440
|
+
prepared[statement.physical_name] = true
|
|
441
|
+
end
|
|
442
|
+
|
|
443
|
+
break if generation == (@prepared_generation || 0)
|
|
444
|
+
end
|
|
445
|
+
end
|
|
446
|
+
|
|
362
447
|
def release_pinned(owner, conn)
|
|
363
448
|
begin
|
|
364
449
|
if conn
|
|
@@ -473,32 +558,27 @@ module PgPipeline
|
|
|
473
558
|
end
|
|
474
559
|
|
|
475
560
|
def select_driver(drivers, rr)
|
|
476
|
-
|
|
477
|
-
|
|
478
|
-
|
|
561
|
+
size = drivers.length
|
|
562
|
+
return [nil, rr] if size.zero?
|
|
563
|
+
|
|
564
|
+
best = nil
|
|
565
|
+
best_index = nil
|
|
566
|
+
best_load = nil
|
|
567
|
+
|
|
568
|
+
size.times do |offset|
|
|
569
|
+
index = (rr + offset) % size
|
|
570
|
+
driver = drivers[index]
|
|
479
571
|
next unless driver.available?
|
|
480
572
|
|
|
481
573
|
load = driver.load
|
|
482
|
-
if
|
|
483
|
-
|
|
484
|
-
|
|
485
|
-
|
|
486
|
-
count += 1
|
|
574
|
+
if best.nil? || load < best_load
|
|
575
|
+
best = driver
|
|
576
|
+
best_index = index
|
|
577
|
+
best_load = load
|
|
487
578
|
end
|
|
488
579
|
end
|
|
489
|
-
return [nil, rr] if count.zero?
|
|
490
580
|
|
|
491
|
-
|
|
492
|
-
index = 0
|
|
493
|
-
drivers.each do |driver|
|
|
494
|
-
next unless driver.available?
|
|
495
|
-
next unless driver.load == min_load
|
|
496
|
-
|
|
497
|
-
return [driver, rr + 1] if index == target
|
|
498
|
-
|
|
499
|
-
index += 1
|
|
500
|
-
end
|
|
501
|
-
[nil, rr]
|
|
581
|
+
best ? [best, (best_index + 1) % size] : [nil, rr]
|
|
502
582
|
end
|
|
503
583
|
|
|
504
584
|
def new_connection(connection_args)
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "errors"
|
|
4
|
+
require_relative "request"
|
|
5
|
+
|
|
6
|
+
module PgPipeline
|
|
7
|
+
class PreparedStatement
|
|
8
|
+
attr_reader :name, :sql, :param_types, :physical_name
|
|
9
|
+
|
|
10
|
+
def initialize(client:, name:, physical_name:, sql:, param_types: nil)
|
|
11
|
+
@client = client
|
|
12
|
+
@name = PreparedStatementOps.snapshot_name(name)
|
|
13
|
+
@physical_name = PreparedStatementOps.snapshot_name(physical_name)
|
|
14
|
+
@sql = RequestOps.snapshot_sql(sql)
|
|
15
|
+
@param_types = RequestOps.snapshot_param_types(param_types)
|
|
16
|
+
freeze
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
def query(params = []) = PreparedStatementOps.query(self, params)
|
|
20
|
+
alias call query
|
|
21
|
+
|
|
22
|
+
def inspect
|
|
23
|
+
"#<#{self.class} name=#{@name.inspect} sql=#{@sql.inspect}>"
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
private
|
|
27
|
+
|
|
28
|
+
attr_reader :client
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
module PreparedStatementOps
|
|
32
|
+
module_function
|
|
33
|
+
|
|
34
|
+
def query(statement, params)
|
|
35
|
+
ClientOps.query_prepared(statement.__send__(:client), statement, params)
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def snapshot_name(name)
|
|
39
|
+
value = name.to_s
|
|
40
|
+
raise ArgumentError, "prepared statement name must not be empty" if value.empty?
|
|
41
|
+
raise ArgumentError, "prepared statement name must not contain NUL" if value.include?("\0")
|
|
42
|
+
|
|
43
|
+
value.frozen? ? value : value.dup.freeze
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
end
|
|
47
|
+
end
|
data/lib/pg_pipeline/request.rb
CHANGED
|
@@ -11,7 +11,7 @@ module PgPipeline
|
|
|
11
11
|
:result, :error
|
|
12
12
|
|
|
13
13
|
def initialize(sql:, params: nil)
|
|
14
|
-
@sql = sql
|
|
14
|
+
@sql = RequestOps.snapshot_sql(sql)
|
|
15
15
|
@params = RequestOps.snapshot_params(params)
|
|
16
16
|
@state = :new
|
|
17
17
|
@condition = Async::Notification.new
|
|
@@ -23,6 +23,14 @@ module PgPipeline
|
|
|
23
23
|
@error = nil
|
|
24
24
|
end
|
|
25
25
|
|
|
26
|
+
def self.prepare(statement) = PrepareRequest.new(statement)
|
|
27
|
+
|
|
28
|
+
def self.prepared_query(statement, params: nil)
|
|
29
|
+
PreparedQueryRequest.new(statement, params: params)
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def operation = :query
|
|
33
|
+
|
|
26
34
|
def queued! = RequestOps.transition!(self, :new, :queued)
|
|
27
35
|
def dispatched! = RequestOps.transition!(self, :queued, :dispatched)
|
|
28
36
|
def accept_result(result) = RequestOps.accept_result(self, result)
|
|
@@ -38,9 +46,37 @@ module PgPipeline
|
|
|
38
46
|
def settled? = @settled
|
|
39
47
|
end
|
|
40
48
|
|
|
49
|
+
class PrepareRequest < Request
|
|
50
|
+
attr_reader :statement_name, :param_types
|
|
51
|
+
|
|
52
|
+
def initialize(statement)
|
|
53
|
+
super(sql: statement.sql)
|
|
54
|
+
@statement_name = RequestOps.snapshot_name(statement.physical_name)
|
|
55
|
+
@param_types = RequestOps.snapshot_param_types(statement.param_types)
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
def operation = :prepare
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
class PreparedQueryRequest < Request
|
|
62
|
+
attr_reader :statement_name
|
|
63
|
+
|
|
64
|
+
def initialize(statement, params: nil)
|
|
65
|
+
super(sql: statement.sql, params: params)
|
|
66
|
+
@statement_name = RequestOps.snapshot_name(statement.physical_name)
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
def operation = :prepared_query
|
|
70
|
+
end
|
|
71
|
+
|
|
41
72
|
module RequestOps
|
|
42
73
|
module_function
|
|
43
74
|
|
|
75
|
+
def snapshot_sql(sql)
|
|
76
|
+
value = sql.to_s
|
|
77
|
+
value.frozen? ? value : value.dup.freeze
|
|
78
|
+
end
|
|
79
|
+
|
|
44
80
|
def snapshot_params(params)
|
|
45
81
|
values = params.nil? ? [] : params
|
|
46
82
|
raise ArgumentError, "params must be an Array" unless values.is_a?(Array)
|
|
@@ -51,16 +87,32 @@ module PgPipeline
|
|
|
51
87
|
def snapshot_value(value)
|
|
52
88
|
case value
|
|
53
89
|
when String
|
|
54
|
-
value.dup.freeze
|
|
90
|
+
value.frozen? ? value : value.dup.freeze
|
|
55
91
|
when Hash
|
|
56
92
|
value.each_with_object({}) do |(key, item), copy|
|
|
57
|
-
copy[key] = item.is_a?(String) ? item.dup.freeze : item
|
|
93
|
+
copy[key] = item.is_a?(String) && !item.frozen? ? item.dup.freeze : item
|
|
58
94
|
end.freeze
|
|
59
95
|
else
|
|
60
96
|
value
|
|
61
97
|
end
|
|
62
98
|
end
|
|
63
99
|
|
|
100
|
+
def snapshot_name(name)
|
|
101
|
+
value = name.to_s
|
|
102
|
+
raise ArgumentError, "statement_name must not be empty" if value.empty?
|
|
103
|
+
|
|
104
|
+
value.frozen? ? value : value.dup.freeze
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
def snapshot_param_types(param_types)
|
|
108
|
+
return nil if param_types.nil?
|
|
109
|
+
raise ArgumentError, "param_types must be an Array or nil" unless param_types.is_a?(Array)
|
|
110
|
+
|
|
111
|
+
param_types.map { |oid| oid.nil? ? nil : Integer(oid) }.freeze
|
|
112
|
+
rescue ArgumentError, TypeError
|
|
113
|
+
raise ArgumentError, "param_types must contain only integer OIDs or nil"
|
|
114
|
+
end
|
|
115
|
+
|
|
64
116
|
def transition!(req, from, to)
|
|
65
117
|
unless req.state == from
|
|
66
118
|
raise ProtocolError, "invalid request transition #{req.state.inspect} -> #{to.inspect}"
|
|
@@ -30,7 +30,11 @@ module PgPipeline
|
|
|
30
30
|
}.freeze
|
|
31
31
|
|
|
32
32
|
def assert_multiplexable!(sql, mode: :default)
|
|
33
|
-
|
|
33
|
+
assert_multiplexable_normalized!(sql, mode: normalize_mode!(mode))
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def assert_multiplexable_normalized!(sql, mode:)
|
|
37
|
+
reason = unsafe_reason_normalized(sql, mode: mode)
|
|
34
38
|
return true unless reason
|
|
35
39
|
|
|
36
40
|
raise UnsafeMultiplexError,
|
|
@@ -43,9 +47,12 @@ module PgPipeline
|
|
|
43
47
|
GUARD_CACHE_LIMIT = 2048
|
|
44
48
|
|
|
45
49
|
def unsafe_reason(sql, mode: :default)
|
|
46
|
-
mode
|
|
50
|
+
unsafe_reason_normalized(sql, mode: normalize_mode!(mode))
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def unsafe_reason_normalized(sql, mode:)
|
|
47
54
|
key = sql.to_s
|
|
48
|
-
cache = guard_cache
|
|
55
|
+
cache = guard_cache.fetch(mode)
|
|
49
56
|
return cache[key] if cache.key?(key)
|
|
50
57
|
|
|
51
58
|
reason = compute_unsafe_reason(key, mode)
|
data/lib/pg_pipeline/version.rb
CHANGED
data/lib/pg_pipeline.rb
CHANGED
|
@@ -7,6 +7,7 @@ require_relative "pg_pipeline/session_guard"
|
|
|
7
7
|
require_relative "pg_pipeline/session"
|
|
8
8
|
require_relative "pg_pipeline/transaction"
|
|
9
9
|
require_relative "pg_pipeline/request"
|
|
10
|
+
require_relative "pg_pipeline/prepared_statement"
|
|
10
11
|
require_relative "pg_pipeline/connection_driver"
|
|
11
12
|
require_relative "pg_pipeline/pool"
|
|
12
13
|
require_relative "pg_pipeline/client"
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: pg_pipeline
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.2.
|
|
4
|
+
version: 0.2.3
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Roman Hajdarov
|
|
@@ -121,6 +121,7 @@ files:
|
|
|
121
121
|
- lib/pg_pipeline/connection_driver.rb
|
|
122
122
|
- lib/pg_pipeline/errors.rb
|
|
123
123
|
- lib/pg_pipeline/pool.rb
|
|
124
|
+
- lib/pg_pipeline/prepared_statement.rb
|
|
124
125
|
- lib/pg_pipeline/request.rb
|
|
125
126
|
- lib/pg_pipeline/server_caps.rb
|
|
126
127
|
- lib/pg_pipeline/session.rb
|