pg_pipeline 0.2.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 5b73820c432250a5622916f522cc7df96e9be218120372d90d30447b2ea4364f
+  data.tar.gz: af6ac07e07a30e7812846c58b56cfb3fcf0722b6fcac99119be4529249a92a0f
+SHA512:
+  metadata.gz: 70249da4404588815fd9b5122ad3bcd564ca8fb8deec8f9068788a27bad9427ebac189bb7ee3ae9f0e05c0461429a00c2a70001749d62f11199777a1d3394472
+  data.tar.gz: b030ef64f18383b8d3fd15177f7375b7347fed99e7e2e2dc387133306a35fccbe775a79aa8545005fef3bffc7d8ed1ba9547a389212de9a3c8801c0558a0d24e

data/CHANGELOG.md

@@ -0,0 +1,156 @@
+# Changelog
+
+## [0.2.0] - unreleased
+
+Initial experimental implementation.
+
+- Single-owner `ConnectionDriver` with separate readiness watchers.
+- Raw nonblocking libpq state machine over ruby-pg.
+- FIFO demultiplexing with request completion only at `PGRES_PIPELINE_SYNC`.
+- Per-unit Sync to isolate independent implicit transactions/error recovery.
+- Real bounded pending queue with shutdown wakeup semantics.
+- Future cancellation drains abandoned protocol slots without backend cancel.
+- Session-neutral multiplexed path plus separate pinned `session` and
+  `transaction` APIs.
+- Pinned connections sanitized with rollback-if-needed + `DISCARD ALL`.
+- Capabilities keyed from `PG.library_version`/protocol version, not server
+  version; libpq 14 minimum, libpq 17 optional fast Sync path.
+- Async-native: Ruby >= 3.3, async ~> 2.42, pg >= 1.5 (chosen for what
+  maximally simplifies the implementation; not held back to an older floor).
+- Opportunistic result draining mirrors libpq's aborted-pipeline readiness
+  corner case instead of relying solely on socket-readable notifications.
+- Strict result-status whitelist; unsupported streaming/COPY modes fail closed.
+- Structured `Client.open` runs inside the caller's Async task instead of
+  creating a hidden task lifetime.
+- Graceful close rejects re-entrant shutdown from inside a pinned block.
+- Deferred `Async::Notification` wakeups avoid re-entrant owner deadlocks on
+  the connection owner; CI now exercises the exact async 2.42.0 dependency floor
+  in addition to the current lockfile-resolved 2.x release.
+- Pinned PostgreSQL 9.3 sessions are physically reconnected after use because
+  pre-9.4 `DISCARD ALL` does not clear sequence `currval`/`lastval` state.
+- Client use is rejected across thread/scheduler boundaries; shared clients are
+  explicitly reactor-local.
+- Connection-loss semantics distinguish requests that were never dispatched from
+  dispatched units whose commit outcome is indeterminate until Sync is observed.
+- String bind values (including the string `:value` in ruby-pg parameter hashes)
+  are snapshotted before asynchronous dispatch to avoid caller-side mutation races
+  while a request is queued; arbitrary custom mutable encoder objects remain the
+  caller's responsibility.
+- Graceful shutdown now counts pinned connection establishment as active work,
+  closing a race where a yielding connect could outlive pool shutdown.
+- Pinned `Session`/`Transaction` handles are fiber-local and expire at block exit,
+  so they cannot race from child tasks or use a recycled connection later.
+- Queue ownership now starts only after `BoundedQueue` actually accepts a Request;
+  a close while a producer is blocked remains definitely pre-dispatch and can
+  fail over to a sibling driver.
+- `NotDispatchedError` retries always use a fresh Request, including when the
+  failed Request was already settled by the losing driver.
+- Timed-out idle health probes can abort only while the probe is still the sole
+  driver work; a concurrent user request makes the probe inconclusive instead
+  of risking an unrelated `IndeterminateResultError`.
+- Client-side bind/encoding exceptions raised before `PQsendQueryParams` are
+  request-local and no longer tear down the shared connection. `PG::UnableToSend`
+  is `NotDispatched`; when the connection remains healthy in pipeline mode that
+  rejection is request-local, while other PG send failures stay driver-fatal.
+  Failure after the query was accepted but before Sync remains indeterminate.
+- CI now separates server-version coverage from source-built client-libpq
+  coverage and exercises the libpq 14 and 16 fallback path explicitly.
+- `reconnect: false` now remains authoritative even when health checks keep the
+  supervisor running; health probing no longer accidentally re-enables reconnect.
+- Unexpected supervisor-loop failures are recorded in `stats[:supervisor_error]`
+  and the loop keeps running (no longer permanently disables reconnect/health).
+- `Transaction#run` / `#savepoint` enforce fiber-local ownership via
+  `SessionOps.ensure_active!`; nested `run` is rejected (`already open`).
+- Nested-run guard errors no longer trigger the outer transaction's
+  `rescue Exception` rollback path (would have ROLLBACK'd the outer BEGIN).
+- `TransactionOps` accesses the physical connection only through
+  `SessionOps.connection` (private `conn` after Session encapsulation).
+
+### Style
+
+- Refactored to a moderate functional-procedural shape: classes are thin state
+  wrappers holding the public API; behavior lives in `*Ops`/module_function
+  modules that receive state explicitly (Caps, DriverOps, PoolOps, RequestOps,
+  SessionOps, TransactionOps, ClientOps, SessionGuard).
+- Implementation comments are kept only where they capture a non-obvious protocol or
+  ownership invariant; routine explanatory noise is omitted.
+- `loop` retained only where idiomatic (BoundedQueue wait-retry); no `while true`.
+
+### Review response (hardening)
+
+- Reconnect: pool supervisor replaces dead pipeline drivers (backoff); reconnect count in stats.
+- BoundedQueue wake-one (no thundering herd under many blocked producers).
+- BoundedQueue transfers a wake when a blocked waiter is cancelled after being
+  signalled, so free capacity cannot strand behind a dead fiber.
+- Observability: client/pool/driver `stats`.
+- Safer defaults: max_in_flight 64, max_pending 256, pinned_size 2.
+- SessionGuard `:strict` mode + documented holes.
+- Transaction savepoints.
+- place_sync stays raw-first (send_pipeline_sync on 17+, sync_pipeline_sync on 14-16) + explicit flush.
+- Falcon per-worker contract, sizing formula, honest example naming (async_app.rb, falcon_config.ru).
+- Packaging: gemspec metadata; CI (unit + PG 14/16/17 matrix); docker-compose.
+- Live integration suite under `spec/integration` (env-gated): multiplex, error
+  isolation, cancel-drain, savepoints, stats, backpressure-on-close, pinned
+  concurrency, driver abort → IndeterminateResultError, reconnect recovery.
+- Unit suite aligned with Ops modules; reconnect/backoff and queue cancel-leak covered.
+
+### Second-pass reliability
+
+- Idle health checks: pool probes idle drivers (SELECT 1 + timeout); timeout
+  aborts only when the probe is still the sole driver work; health_failures in stats.
+- abort! cancels in-flight pinned work via CancelRequest.
+- Submit failover: fresh Request per attempt; `NotDispatchedError` can retry
+  even when the losing Request was already settled.
+- Strict guard precise denylist (no pg_typeof/setup/xact-advisory false positives).
+- CI separates server-major coverage from source-built client-libpq 14/16 coverage.
+- benchmarks/ (fiber-storm latency, multi-worker connection-count smoke).
+- Pool driver slot arrays self-heal size (reconnect/health safe under partial init).
+- Unit coverage: failover, health_probe, pinned cancel on abort.
+
+- Pinned recycle rechecks shutdown after yielding reset/reconnect work so a
+  connection cannot be returned to the free list after pool shutdown.
+- Replacement pipeline drivers are children of the pool's original parent task,
+  not the supervisor; ownership transfer is explicit so cancellation cannot orphan
+  a just-created replacement.
+- `Session` no longer exposes its raw `PG::Connection`; Ruby-side typemap/callback
+  mutations cannot leak through a recycled pinned handle.
+- `Transaction` lifecycle (`BEGIN`/`COMMIT`/`ROLLBACK`) is internal; callers cannot
+  recursively invoke the outer transaction runner or mutate its open state.
+- `Client` lifecycle/reactor ownership state is private, preventing callers from
+  disabling close or bypassing the thread/scheduler guard.
+- Timing options reject negative, NaN and infinite values; a closed Pool cannot be
+  restarted; unknown guard modes fail closed.
+- CI YAML is valid, Linux platforms are present in the lockfile, and source-built
+  jobs assert the actual linked libpq 14/16 versions rather than confusing server
+  version coverage with client capability coverage.
+- Documented upstream ruby-pg 1.6.3 parameter-encoding overflow (#719); its fix is
+  merged upstream but no newer RubyGems release is available as of July 2026.
+- Documented the ordinary pinned-transaction commit-acknowledgement ambiguity: a
+  transport failure around `COMMIT` is not proof that the transaction rolled back.
+- Re-entrant pinned checkout from the same task now fails immediately instead of
+  deadlocking at `pinned_size: 1` or silently using a second independent session.
+- Supervisor wake cadence now respects a shorter health interval independently of
+  reconnect cadence; `health_interval: 0` keeps probe-on-each-supervisor-cycle
+  semantics without introducing a busy loop.
+- Driver load accounting includes the narrow dispatching state so routing and
+  health checks cannot observe a transient false-idle driver.
+- Pinned recycle cleanup now treats `Async::Cancel < Exception` as a cleanup
+  boundary: cancellation during rollback/sanitize/reconnect closes the physical
+  connection instead of orphaning it outside the pool. Graceful close also closes
+  already-idle pinned sockets before waiting for active checkouts.
+
+## 0.2.1 (perf + measurement)
+
+- SessionGuard: bounded per-SQL memo cache for unsafe_reason. Repeated SQL
+  (parameterized queries) now costs ~0 allocations instead of ~330 objects/call
+  (measured); eliminates the code_only/byteslice line that showed up in profiles.
+- Client#query: drop redundant sql dup (Request owns the one immutable copy).
+- PoolOps.select_driver: allocation-free least-loaded scan (was 2-3 arrays/call);
+  behavior verified identical across 20k random cases.
+- bench_kit/metrics.rb + rake bench:metrics: full picture in one run —
+  allocations/query + GC pressure, RubyProf process_time (CPU) and allocations,
+  RubyProf wall (sanity), and StackProf cpu/wall sampling. Asserts run outside
+  every profiler; warmup keeps connection setup out of frame.
+- profile_ci_scenario.rb: correctness asserts moved out of the profiled block;
+  GC.start before profiling.
+- Dev dependency: stackprof (~> 0.2).

data/DESIGN.md

@@ -0,0 +1,298 @@
+# pg_pipeline design
+
+## 1. Scope
+
+`pg_pipeline` is a driver-adjacent Ruby control plane over `ruby-pg`. It does not
+parse the wire protocol and does not replace libpq.
+
+The target workload is many concurrent Async/Falcon fibers issuing independent
+small/medium PostgreSQL operations where RTT and/or connection count matter.
+Pipeline mode does not parallelize execution inside one PostgreSQL backend.
+
+## 2. Non-negotiable ownership invariant
+
+```text
+one ConnectionDriver == one PG::Connection == one task that calls it
+```
+
+The owner task is the only code allowed to call `PG::Connection` methods.
+Reader/writer watcher tasks only wait on the memoized socket IO and enqueue
+readiness events.
+
+This avoids concurrent mutation of libpq protocol state while still giving the
+owner three wakeup sources:
+
+```text
+submitted request ----\
+socket readable -------+--> event queue --> OWNER --> PG::Connection
+socket writable -------/
+```
+
+## 3. Raw nonblocking mode
+
+At startup the driver calls:
+
+```ruby
+conn.setnonblocking(true)
+conn.enter_pipeline_mode
+```
+
+In ruby-pg this means the application takes responsibility for send/flush
+blocking states. The owner uses raw primitives where the distinction matters:
+
+```text
+send_query_params
+sync_flush
+consume_input
+is_busy
+sync_get_result
+```
+
+`is_busy` is only a readiness predicate: false means the next
+`sync_get_result` will not block. It is not a request boundary.
+
+The owner also checks `is_busy` opportunistically before going back to sleep,
+not only after a socket-readable notification. PostgreSQL's own
+`libpq_pipeline.c` does the same because libpq can synthesize an aborted-pipeline
+result locally without making the socket newly readable.
+
+## 4. Protocol/FIFO model
+
+Each multiplexed unit is encoded as:
+
+```text
+extended-protocol query
+Sync
+```
+
+The request is appended to the in-flight FIFO only after both operations have
+been queued successfully.
+
+Results are associated with `@inflight.first`:
+
+- ordinary/error `PG::Result` -> belongs to the FIFO front;
+- `nil` -> end of that query's results;
+- `PGRES_PIPELINE_SYNC` -> end of that logical unit; only here is FIFO front
+  removed and its waiter resolved;
+- `PGRES_PIPELINE_ABORTED` -> record an error for the front, but do **not** pop
+  it until its Sync.
+
+A server-side SQL error is therefore request-local. A client-side send/flush/
+protocol error is connection-fatal because the driver can no longer prove how
+its local FIFO maps to libpq's internal command queue.
+
+The v1 driver accepts only the normal whole-result statuses
+`EMPTY_QUERY`/`COMMAND_OK`/`TUPLES_OK`. COPY, single-row/chunked-row statuses and
+unknown statuses are protocol errors because the public API does not enable
+those result modes.
+
+## 5. Why Sync is per unit
+
+Without an explicit transaction, commands between synchronization points form an
+implicit transaction/error-recovery segment. Sharing one trailing Sync across
+independent callers would couple their commit/error semantics.
+
+Therefore v1 always uses one Sync per independent unit. On libpq 17+ the driver
+uses `PQsendPipelineSync`; on older libpq it uses `PQpipelineSync`. In both cases
+it explicitly checks `PQflush` afterwards because nonblocking output can remain
+buffered.
+
+## 6. Shared-session contract
+
+Autocommit alone is not sufficient. Multiplexed callers share a PostgreSQL
+session. `SET`, `SET ROLE`, prepared statements, temp objects, LISTEN state and
+session advisory locks can leak across callers.
+
+`Client#query` therefore has a **session-neutral SQL contract**. `SessionGuard`
+is intentionally conservative and catches common hazards, but it cannot prove
+that arbitrary SQL or user-defined functions are session-neutral.
+
+Anything stateful goes through an exclusive pinned connection:
+
+- `Client#session` — no implicit BEGIN;
+- `Client#transaction` — BEGIN/COMMIT/ROLLBACK.
+
+Pinned connections are opened lazily up to `pinned_size`; a query-only process
+does not reserve an otherwise idle second pool at startup. Before a pinned
+physical connection is reused, the pool makes sure it is outside an explicit
+transaction. PostgreSQL 9.4+ is sanitized with `DISCARD ALL`; PostgreSQL 9.3 is
+physically reconnected because its `DISCARD ALL` cannot clear sequence state.
+
+`Session`/`Transaction` handles are fiber-local scoped capabilities. They may
+only be used by the fiber that received them and are invalidated when their block
+exits. An escaped object therefore cannot race another fiber or keep using a
+physical connection after the pool has handed it to another caller.
+
+Calling `Client#close` from inside its own `session`/`transaction` block is
+rejected instead of deadlocking while graceful shutdown waits for that same
+pinned connection to be returned. A task that already owns a pinned checkout
+also cannot recursively call `Client#session`/`Client#transaction`; that would
+deadlock with `pinned_size: 1` or create a misleading second session with a
+larger pool. Nested transactional work uses `Transaction#savepoint` instead.
+
+## 7. Cancellation
+
+There are two unrelated concepts:
+
+1. **waiter cancellation** — caller stops caring about its Future/result;
+2. **backend cancellation** — PostgreSQL CancelRequest cancels the statement
+   currently executing on that backend.
+
+Only (1) exists on the multiplexed API. An abandoned request already sent to the
+server remains in the in-flight FIFO and is fully drained to Sync. Backend query
+cancellation would be unsafe without a dedicated/pinned connection or stronger
+knowledge that the target is the currently executing FIFO head. A cancelled
+waiter therefore does **not** mean a mutating statement was cancelled; it can
+still execute and commit.
+
+A connection failure before dispatch is reported as `NotDispatchedError`. A
+`PG::UnableToSend` from `PQsendQueryParams` is also definitely pre-dispatch; if
+libpq still reports a healthy connection in pipeline mode, that rejection stays
+request-local instead of poisoning unrelated in-flight units. Other `PG::Error`
+send failures are treated as driver-fatal. Once a request has entered
+dispatch/in-flight state, losing the connection before its `PGRES_PIPELINE_SYNC`
+is observed is reported as `IndeterminateResultError`: for mutations the
+server-side commit outcome is unknowable and automatic retry is unsafe without
+an application-level idempotency strategy.
+
+Pinned `Client#transaction` has the standard PostgreSQL commit-acknowledgement
+ambiguity as well: a transport failure during `COMMIT` does not prove rollback
+or commit. That path intentionally preserves the underlying `PG::Error`; callers
+that retry non-idempotent transactions need application-level idempotency or
+reconciliation rather than assuming the transaction did not commit.
+
+## 8. Backpressure and shutdown
+
+`max_pending` is enforced by a small reactor-local bounded queue. On async 2.42
+`Async::LimitedQueue` (with `#close`) would cover plain bounded backpressure; the
+in-tree `BoundedQueue` is kept only for the two things it adds and the failure
+model needs: a **typed** close (a producer blocked on backpressure during an
+abort is woken with `NotDispatchedError`, not a generic closed error) and
+`#drain` (hand back queued-but-undispatched requests to reject in one shot).
+
+`max_in_flight` bounds the number of dispatched pipeline units.
+
+Graceful shutdown:
+
+```text
+stop accepting -> reject/wake blocked producers -> drain queued/in-flight work
+-> flush -> exit pipeline mode -> stop watchers -> close socket
+```
+
+Fatal shutdown:
+
+```text
+stop accepting -> fail every owned request -> stop watchers -> close socket
+```
+
+## 9. Pinned connection hygiene
+
+Pinned callers are allowed to mutate session state. Reusing such a connection
+without reset would leak that state to the next caller. After each pinned block:
+
+1. if transaction status is `INTRANS`/`INERROR`, issue `ROLLBACK`;
+2. require an idle connection;
+3. on PostgreSQL 9.4+, run `DISCARD ALL`;
+4. on PostgreSQL 9.3, reconnect instead because pre-9.4 `DISCARD ALL` does not
+   clear sequence `currval`/`lastval` state;
+5. if cleanup fails, close and replace the physical connection;
+6. if replacement also fails, mark the pinned pool broken rather than reusing a
+   contaminated connection.
+
+## 10. Deferred fiber wakeups
+
+Immediately resuming a waiter from inside the connection owner is unsafe: a
+resumed caller could re-enter the driver (submit, or close the client and wait on
+the owner) while the owner is mid-`signal`. Internal wait/notify points therefore
+use `Async::Notification`, which resumes waiters on a later reactor turn, avoiding
+reentrancy into the owner. The same rule applies to request completion,
+bounded-queue backpressure and pinned-pool idle notifications.
+
+> Note: the supported Async range starts at 2.42 and stays below 3. The lockfile
+> exercises the current compatible 2.x release, while CI separately runs the unit
+> suite against the exact 2.42.0 floor so this coordination behavior is not merely
+> assumed from a broad pessimistic dependency range.
+
+## 11. Version policy
+
+| Component | Minimum | Reason |
+|---|---:|---|
+| Ruby | 3.3 | our dev-env floor; required by the current async line |
+| async | ~> 2.42 | Async-native; floor 2.42.0 is exercised separately in CI |
+| pg | >= 1.5, < 2 | pipeline bindings + raw result/flush + setnonblocking |
+| libpq | 14 | pipeline API introduced here (client-side) |
+| server | protocol v3 | pipeline is client-side; NO server-14 requirement |
+
+We go Async-native and depend on the current async line rather than reinventing
+its coordination primitives to hold an older Ruby floor. Ruby/async are OUR
+dev-env floors — chosen freely for what maximally simplifies the implementation.
+libpq/server are the USER's floor and are NOT raised without cause: libpq 14 is
+the true minimum (no pipeline mode below it); the server only needs protocol v3.
+libpq 17 is an optional runtime-detected fast path (`PQsendPipelineSync`,
+chunked-rows, non-blocking cancel), never a floor.
+
+> The server floor is the user's environment; revisit annually against
+> PostgreSQL's EOL calendar. Capability gating is by `PG.library_version`
+> (the client libpq), never by `conn.server_version`.
+
+## 12. Known limitations before production
+
+- Pipeline driver death is recovered by the pool supervisor (`reconnect: true`
+  default) with exponential backoff; while a slot is down traffic goes to live
+  drivers, and with zero live drivers `#query` raises `NotDispatchedError`.
+  Idle drivers are also probed (`health_check: true`) with `SELECT 1`. A
+  timed-out probe may abort the connection only while it remains the sole
+  driver work; concurrent user work makes the probe inconclusive instead of
+  allowing a background check to make that work indeterminate.
+- No PostgreSQL query cancellation on the **multiplexed** path (would target
+  whichever statement is currently executing on the shared backend).
+- `abort!` sends `CancelRequest` to checked-out **pinned** connections
+  (`cancel_pinned_on_abort: true`); it does not force-close the socket if the
+  backend ignores cancel.
+- No streaming/single-row/chunked-row result API.
+- No cross-thread/reactor sharing; `Client` rejects use from a different thread or Fiber scheduler.
+- `SessionGuard` is policy/ergonomics, not a security boundary. `:strict` adds a
+  precise denylist (`nextval`/`setval`/`pg_export_snapshot`, …) without the old
+  broad `pg_*(` / `set*(` false positives; UDFs can still mutate session state.
+- Live integration covers **server** PG 14/16/17. CI prints the linked **client**
+  libpq version, but does not yet matrix distinct client-libpq majors (one
+  runner libpq across all server jobs). Pipeline capability is client-side, so
+  a dedicated libpq 14/16/17 client matrix remains future work.
+- On libpq 14–16, `place_sync` uses `PQpipelineSync` / ruby-pg `sync_pipeline_sync`
+  (flush coupled); libpq 17+ uses `send_pipeline_sync` + explicit `sync_flush`.
+- Pinned recycle still runs full `DISCARD ALL` (no lighter reset profile / recycle
+  latency metric yet).
+- The latest released ruby-pg (1.6.3 as of July 2026) has upstream issue #719
+  in very large query-parameter encoding. The fix is merged upstream but no newer
+  RubyGems release exists yet; untrusted multi-gigabyte parameter sets must be
+  bounded or deployments should build ruby-pg from a revision containing the fix.
+
+## 13. Roadmap status (review response)
+
+Implemented:
+
+- Automatic reconnect / replacement of dead pipeline drivers (pool supervisor
+  with exponential backoff); `client.stats[:reconnects]`.
+- Idle health checks on zero-load drivers; `client.stats[:health_failures]`.
+- BoundedQueue wake-one + cancel wake-transfer (no thundering herd / stranded slots).
+- Observability: public `client.stats` plus internal per-driver `stats`.
+- Safer fiber-storm defaults: `max_in_flight: 64`, `max_pending: 256`,
+  `pinned_size: 2`.
+- `SessionGuard` `:strict` precise denylist + documented holes.
+- Savepoints (`Transaction#savepoint`).
+- Submit failover: `NotDispatchedError` retries on a **new** Request against a
+  live sibling; the failed Request object itself is never re-used.
+- `abort!` CancelRequest on in-use pinned connections.
+- Falcon per-worker contract + sizing formula; reactor-local Client fail-fast.
+- Packaging: gemspec metadata, CI (unit + PG server 14/16/17), docker-compose,
+  examples, `benchmarks/pipeline_bench.rb` + `benchmarks/multiworker_smoke.rb`.
+- Live integration suite (multiplex, isolation, cancel-drain, savepoints, stats,
+  backpressure, pinned concurrency, indeterminate on abort, reconnect, backend
+  terminate recovery, pinned abort cancel).
+
+Still open:
+
+- Dedicated client-libpq 14/16/17 CI matrix (not only server majors).
+- Lighter pinned reset / recycle-latency metrics under high-TPS tx.
+- Force socket close if pinned cancel is ignored.
+- Published bench numbers in README; tighten multiworker smoke to peak occupancy.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,183 @@
+# pg_pipeline
+
+PostgreSQL pipeline multiplexing for Ruby — inspired by [tokio-postgres](https://github.com/sfackler/rust-postgres).
+
+## What is PostgreSQL pipeline mode?
+
+Normally every query follows a request/reply cycle:
+
+```
+fiber → SEND query → wait RTT → RECV result → done
+```
+
+With [libpq pipeline mode](https://www.postgresql.org/docs/current/libpq-pipeline-mode.html) (libpq ≥ 14)
+the client can send multiple queries without waiting for each result first:
+
+```
+fiber A → SEND query A ─┐
+fiber B → SEND query B ─┤─── one RTT ───┐
+fiber C → SEND query C ─┘               ├─ RECV A, B, C
+                                         └─ route back to fibers
+```
+
+At 10 ms RTT a naïve pool does ~100 queries/s per connection.  
+A pipelined connection can do thousands — the wire stays full instead of sitting idle.
+
+`pg_pipeline` is the Ruby control plane that does exactly this: it multiplexes
+independent queries from many Async fibers onto a small pool of libpq connections,
+routes FIFO results back to the right fiber, and keeps transactional/session work
+on separate pinned connections. All wire protocol work stays in libpq; zero C code here.
+
+The single-owner connection model, FIFO result ownership, and lifecycle approach
+are directly inspired by tokio-postgres.
+
+## Installation
+
+```ruby
+# Gemfile
+gem "pg_pipeline"
+```
+
+Requires Ruby ≥ 3.3, `async ~> 2.42`, `pg ≥ 1.5`, and **libpq ≥ 14** at runtime.
+
+## Usage
+
+### Multiplexed queries
+
+```ruby
+require "pg_pipeline"
+
+Sync do |task|
+  PgPipeline::Client.open(ENV["DATABASE_URL"]) do |db|
+    # 20 queries fly out in parallel over a handful of connections
+    results = (1..20).map do |id|
+      task.async { db.query("SELECT $1::int AS id, now() AS at", [id]) }
+    end.map(&:wait)
+
+    p results.first # => [{"id"=>"1", "at"=>"..."}]
+  end
+end
+```
+
+`Client.open` manages the pool lifecycle. `db.query` is fiber-safe and multiplexed —
+every fiber yields while waiting, the event loop stays unblocked.
+
+### Transactions
+
+```ruby
+db.transaction do |tx|
+  tx.exec("INSERT INTO orders(user_id, total) VALUES ($1, $2)", [user_id, total])
+  tx.exec("UPDATE balances SET reserved = reserved + $1 WHERE user_id = $2", [total, user_id])
+end
+# COMMIT on success, ROLLBACK on any exception
+```
+
+Nested atomicity with savepoints:
+
+```ruby
+db.transaction do |tx|
+  tx.exec("INSERT INTO events(kind) VALUES ('started')", [])
+  tx.savepoint do |sp|
+    sp.exec("INSERT INTO risky_table(x) VALUES ($1)", [val])
+    # raises → rolls back to savepoint only, outer tx survives
+  end
+end
+```
+
+### Session (DDL, LISTEN, SET, prepared statements)
+
+```ruby
+db.session do |s|
+  s.exec("SET application_name = 'worker-1'")
+  s.exec("PREPARE by_id AS SELECT * FROM users WHERE id = $1")
+  rows = s.exec_prepared("by_id", [42])
+end
+```
+
+The session gets an exclusive pinned connection. It's automatically cleaned up
+(`DISCARD ALL`) before the connection returns to the pool.
+
+### Pool configuration
+
+```ruby
+PgPipeline::Client.open(
+  ENV["DATABASE_URL"],
+  pipeline_size: 4,   # pipelined connections per client
+  pinned_size:   2,   # exclusive connections for session/transaction
+  guard:         :strict
+) do |db|
+  # ...
+end
+```
+
+Total PostgreSQL connections per process: `pipeline_size + pinned_size`.
+
+### Observability
+
+```ruby
+db.stats
+# => {
+#   pipeline: { size: 4, live: 4, drivers: [{load: 3, in_flight: 2, ...}, ...] },
+#   pinned:   { size: 2, active: 1, free: 1 },
+#   reconnects: 0
+# }
+```
+
+`load` (pending + in_flight) per driver is the head-of-line signal.
+
+## Failure model
+
+| Error | Meaning | Retry safe? |
+|---|---|---|
+| `NotDispatchedError` | query never reached the wire | ✅ yes |
+| `IndeterminateResultError` | query was sent, Sync not observed | ⚠️ only if idempotent |
+| `UnsafeMultiplexError` | session-mutating SQL on multiplexed path | — fix the call site |
+
+Cancelling a fiber does **not** send `CancelRequest` to PostgreSQL — the query may
+still execute. For mutations, do not retry blindly after a timeout.
+
+## Falcon integration
+
+One `Client` per worker, created after fork:
+
+```ruby
+# falcon.rb / config.ru
+app = Rack::Builder.new do
+  use MyMiddleware
+  run MyApp
+end
+
+Async do
+  db = PgPipeline::Client.new(ENV["DATABASE_URL"]).start
+  run app  # fibers in this worker share db
+end
+```
+
+See `examples/falcon_config.ru` for a complete setup.
+
+## Benchmark
+
+```bash
+docker compose up -d pg17
+export PG_PIPELINE_URL=postgres://postgres:postgres@127.0.0.1:5417/postgres
+
+bundle exec rake bench:throughput   # fibers/s, p99
+bundle exec rake bench:ab           # pipeline vs naive pool
+```
+
+For realistic numbers inject latency first:
+
+```bash
+# terminal 1
+bundle exec rake bench:proxy RTT_MS=10 UPSTREAM=127.0.0.1:5417
+
+# terminal 2
+PG_PIPELINE_URL=postgres://postgres:postgres@127.0.0.1:6432/postgres \
+  bundle exec rake bench:ab
+```
+
+At 10 ms RTT, pipeline mode delivers ~7× better p99 with 8× fewer server connections.
+
+## License
+
+MIT