pipeloader 0.0.1 → 0.0.3

data/README.md

@@ -1,40 +1,119 @@
 # pipeloader
 
-Transparent libpq pipelining for **graphql-ruby on ActiveRecord**. During GraphQL
-response building, every ActiveRecord `SELECT` is routed through a libpq pipeline,
-so a nested query resolves in roughly **one round trip per tree level** — with
-**plain resolvers and plain models**. No Futures, no `dataloader.load`, no field
-changes.
+Cut ActiveRecord N+1 on both axes, round trips and query count, with plain models and no
+`dataloader.load` keys. The pieces compose:
+
+- `use Pipeloader` routes every `SELECT` in a [graphql-ruby](#adopting-it) response through
+  a libpq pipeline, so a nested query resolves in roughly one round trip per tree level.
+  Plain resolvers, no Futures, no field changes.
+- `auto_fuse` collapses each level's per-record lookups into one `WHERE key = ANY($1)`,
+  dropping query count to DataLoader's.
+- [`Pipeloader::Batch`](#batch-loaders-for-plain-activerecord) brings that batching to plain
+  ActiveRecord (`batch_has_many`, `batch_belongs_to`) for jobs, serializers, and other
+  non-GraphQL paths.
+
+Adopt them together: a fused or batch-loaded query is itself pipelined, so you get
+DataLoader-class query counts and one round trip per level at once.
 
 ## Adopting it
 
-One line:
+These compose; adopt the ones your app needs. A query gathered by fusion or a batch loader
+is itself pipelined, so you get few queries and few round trips together.
+
+### Pipelining
+
+One line. Types and resolvers stay ordinary ActiveRecord:
 
 ```ruby
 class AppSchema < GraphQL::Schema
   use Pipeloader
 end
+
+class Types::Post < GraphQL::Schema::Object
+  field :title, String, null: false
+  field :author, Types::Author, null: false       # resolves via post.author
+  field :comments, [Types::Comment], null: false   # resolves via post.comments
+end
 ```
 
-That's the whole adoption surface. Your types and resolvers stay exactly as they
-are — ordinary ActiveRecord:
+Any AR SELECT issued while building the response (a `belongs_to`, a `has_many`, a
+`.where(...)` in a hand-written resolver) is intercepted and pipelined. It hooks AR's
+query path rather than the GraphQL field, so nothing leaks back to synchronous N+1, even
+from custom resolver code. By default the pipeline fetches whole rows.
+
+### Field-exact projection (opt-in)
+
+Set `field_exact` and each SELECT narrows to the columns the query selected, using
+graphql-ruby's `lookahead`:
 
 ```ruby
+Pipeloader.field_exact = true            # globally, before your types load, or
+
 class Types::Post < GraphQL::Schema::Object
+  pipeloader_field_exact!                # per type
   field :title, String, null: false
-  field :author, Types::Author, null: false       # resolves via post.author
-  field :comments, [Types::Comment], null: false   # resolves via post.comments
+  field :author, Types::Author, null: false
+end
+```
+
+For `{ posts { title author { name } } }` the posts SELECT becomes
+`SELECT id, title, author_id FROM ...` (primary key, selected column, and the FK needed
+for `author`), and the authors SELECT becomes `SELECT id, name FROM ...`.
+
+Projection narrows only when it can prove every selected field reads a known column or
+association. If a selection is opaque (a computed field, a custom resolver, anything it
+can't map to a column) it falls back to a whole-row fetch for that record, so a projected
+field never raises `MissingAttributeError`. A computed field can declare the columns it
+reads with `selects:`, so projection keeps them:
+
+```ruby
+field :excerpt, String, null: false, selects: %i[body]
+def excerpt = object.body[0, 200]
+```
+
+With no opt-in, `selects:` is accepted and ignored and every SELECT is whole-row.
+
+### Auto-fuse (opt-in)
+
+Field-exact also fuses: the per-record `belongs_to` / `has_one` / `has_many` lookups on a
+level collapse into one `WHERE key = ANY($1)` (DataLoader-class server cost, still
+pipelined, so round trips stay at the tree depth). To get that fusion whole-row, with no
+projection and no resolver code, set `auto_fuse`:
+
+```ruby
+Pipeloader.auto_fuse = true              # before your types load
+```
+
+A plain `object.author` / `object.comments` now fuses automatically. It fuses only when
+the demux is provably unambiguous (a unique primary key for `belongs_to`, a unique FK
+index for `has_one`, a bare unscoped `has_many`). Anything else (scopes, chained
+`order`/`limit`, polymorphic, custom resolvers, SQLite) falls back to the plain pipelined
+load. Results are byte-identical to the un-fused path.
+
+### Batch loaders
+
+The same gathering for plain ActiveRecord, for the jobs, serializers, and non-GraphQL
+endpoints the resolvers don't cover. Include the concern and swap `has_many` for
+`batch_has_many` (or `belongs_to` for `batch_belongs_to`):
+
+```ruby
+class Author < ApplicationRecord
+  include Pipeloader::Batch::Model
+  batch_has_many :books
 end
+
+Author.all.to_a.each { |a| a.books.to_a }   # one query for everyone's books
 ```
 
-`post.author`, `post.comments`, a `has_many`, a `.where(...)` in a hand-written
-resolver — any AR SELECT issued while building the response is intercepted and
-pipelined. Because it hooks AR's query path (not the GraphQL field), nothing
-leaks back to synchronous N+1, even from custom resolver code.
+`a.books` loads for every author loaded alongside it on first access, as one `IN` query via
+AR's `Preloader`, with no setup: the sibling group is stamped onto the records as they load.
+Inside a `use Pipeloader` response those batched queries are pipelined too. Full surface (the
+chainable proxy, counts and aggregates, the general `batch` macro) in
+[Batch loaders for plain ActiveRecord](#batch-loaders-for-plain-activerecord) below.
 
 ## What it does
 
-`example/run.rb` — plain resolvers, against a seeded database:
+`example/run.rb`, plain resolvers against a seeded database:
 
 ```
 { posts(limit: 50) { title author { name } comments { body commenter { name } } } }
@@ -45,149 +124,140 @@ queries pipelined:    403
 naive N+1 would be:   ~594 round trips
 ```
 
-Three round trips: `posts` → (`authors` + `comments`) → `commenters`. The to-one
-`author` and the to-many `comments` are *different shapes at the same level*, yet
-collapse into a single round trip.
+Three round trips: `posts`, then `authors` and `comments`, then `commenters`. The to-one
+`author` and the to-many `comments` are different shapes at the same level but collapse
+into one round trip.
+
+## How it works
+
+1. `use GraphQL::Dataloader` runs resolution in fibers, so a synchronous-looking
+   `post.author` can yield instead of blocking and sibling queries gather before anything
+   hits the wire.
+2. A monkey-patch on `select_all` hands each SELECT to a Dataloader source instead of
+   running it. The active dataloader is stashed on the connection for the multiplex (and
+   cleared after), so the patch finds it as `self`.
+3. When the fibers park, the source prepares each distinct query shape (once per request,
+   reused across bursts), Bind/Executes every gathered query in one libpq burst
+   (`enter_pipeline_mode` to `pipeline_sync`), and returns an `ActiveRecord::Result` per
+   query so AR builds models normally.
+
+Prepared statements are scoped to the request. The next request's first burst
+`DEALLOCATE`s the previous one's, piggybacked into the same pipeline so cleanup costs no
+extra round trip, so no plan goes stale across a reconnect or migration. If a query
+errors, the burst is drained to its sync point, the connection is restored, and the error
+is raised rather than swallowed.
 
 ## Benchmark
 
-The same 3-level query (`posts → author + comments → commenter`, 25 posts),
-resolved four ways, with **real** network latency added by a local TCP proxy in
-front of Postgres (`example/latency_proxy.rb` delays the request direction, so a
-synchronous query pays RTT once and a pipelined burst pays it once). Min of 3
-iterations; your numbers will vary.
-
-| approach | RTT 0 | RTT 1 ms | RTT 5 ms | round-trips |
-|---|--:|--:|--:|--:|
-| naive (N+1) | 94 ms | 505 ms | **1972 ms** | 290 |
-| AR `includes` (hand-written) | 17 ms | 22 ms | 42 ms | 4 |
-| `GraphQL::Dataloader` | 16 ms | 21 ms | 42 ms | 4 |
-| **pipeloader** | 41 ms | 45 ms | **73 ms** | **3** |
-
-Reading it honestly:
-
-- **vs the N+1 you actually have** — the headline. pipeloader turns 290 round
-  trips into 3 with zero resolver code, so at a 5 ms hop it's **~24× faster** than
-  naive. Most "there's an N+1 in here somewhere" code is the naive row.
-- **vs batching (`includes` / `GraphQL::Dataloader`)** — at low/moderate RTT,
-  batching still wins: its 4 `IN` queries do less work than pipeloader's ~400
-  prepared point queries. pipeloader prepares + caches statements per connection
-  (so parse cost is amortized to ~one parse per query shape), but it still runs
-  400 bind+executes and builds 400 results. **Pipelining cuts round trips;
-  batching cuts server work.** pipeloader does fewer round trips (3 vs 4 — it
-  collapses the to-one `author` and to-many `comments` into one burst, where
-  Dataloader runs them as two sequential sources), so it closes the gap as RTT
-  rises and passes the batchers around ~25 ms RTT (cross-region). Same
-  point-vs-batch tradeoff the Go experiments in this repo show.
-- **What pipeloader actually buys you: zero code, for any query shape.**
-  `GraphQL::Dataloader` needs a source plus a `.load` call per association;
-  `includes` must be hand-written per query and kept in sync with the selection.
-  pipeloader is `use Pipeloader` and ordinary resolvers.
-
-Run it: `ruby example/bench.rb` (needs the seeded `graphql_experiment` DB).
-
-### Scaling with tree shape
-
-That benchmark is a *narrow* tree (3 deep, 2 relations at its widest), which is
-close to the worst case for pipeloader. The gap widens with **width**, because:
-
-- **pipeloader round trips = tree depth** — one burst per level, any width.
-- **batching round trips = Σ (distinct target tables per level)** — each is its
-  own `IN` query (a Dataloader source, or an `includes` preload).
-
-A *wide* query — issues fanning out to assignee, creator, project, parent, and
-comments, those nesting to team, lead, and authors (`example/bench_wide.rb`):
-
-| approach | RTT 0 | RTT 1 ms | RTT 5 ms | round-trips |
-|---|--:|--:|--:|--:|
-| naive (N+1) | 63 ms | 278 ms | 1115 ms | 164 |
-| AR `includes` | 13 ms | 29 ms | 91 ms | 11 |
-| `GraphQL::Dataloader` | 9 ms | 20 ms | 57 ms | 7 |
-| **pipeloader** | 28 ms | 34 ms | **51 ms** | **3** |
-
-pipeloader's round trips stay at **3** (the depth) while batching climbs to 7–11,
-so at a 5 ms hop **pipeloader is the fastest of all** — the point-vs-batch
-crossover dropped from ~25 ms (narrow) to under 5 ms (wide). The wider and deeper
-the tree, the lower the RTT at which pipelining wins, because pipelining is the
-only one of the three whose round trips don't grow with the query.
+A wide GraphQL query (10 issues, each fanning out to assignee, creator, project, parent,
+and comments, those nesting to team, lead, and authors), resolved against Postgres at a
+realistic 5 ms network RTT (app and primary DB in different AZs through a pooler) via a
+local TCP proxy. Min of 3 iterations; your numbers will vary.
 
-## How it works
+| approach | time | round-trips |
+|---|--:|--:|
+| naive (N+1) | 1160 ms | 164 |
+| AR `includes` (hand-written) | 83 ms | 11 |
+| `GraphQL::Dataloader` | 56 ms | 7 |
+| pipeloader | 62 ms | 3 |
+| **pipeloader (`auto_fuse`)** | **46 ms** | **3** |
 
-1. **`use GraphQL::Dataloader`** — runs resolution in fibers. This is what lets a
-   synchronous-looking `post.author` *yield* instead of blocking, so sibling
-   queries can gather before anything hits the wire.
-2. **A monkey-patch on `select_all`** — while a response is being built, AR's
-   SELECT path hands the query to a Dataloader source instead of executing it.
-   The active dataloader is **stashed on the connection** for the duration of the
-   multiplex (and cleared at the end), so the patch finds it as `self`.
-3. **The source pipelines** — when the fibers all park, it prepares each distinct
-   SQL once (cached per connection), then sends every gathered query as one libpq
-   burst (`enter_pipeline_mode` … `pipeline_sync`), reads the results, and returns
-   an `ActiveRecord::Result` per query so AR builds models normally.
+Against the N+1 you have, pipeloader turns 164 round trips into 3 with no resolver code,
+about 25x faster than naive at this latency.
 
-## Field-exact projection (opt-in)
+Against batching, latency multiplies round trips, and pipeloader does the fewest: 3, the
+tree depth, where `includes` and `GraphQL::Dataloader` run a separate `IN` query per
+association (7 to 11). The transparent path does more server work (N point queries) for
+those few round trips and lands close to Dataloader. `auto_fuse` fuses each level into one
+`WHERE key = ANY($1)`, getting Dataloader's server work and pipeloader's round trips, and
+comes out fastest.
 
-By default AR picks the columns (`SELECT *`), which keeps adoption zero-effort. If
-you want the pipeline to fetch **only the columns the query selected**, opt in and
-pipeloader narrows each SELECT using graphql-ruby's `lookahead`:
+`GraphQL::Dataloader` needs a source and a `.load` per association; `includes` is
+hand-written per query. pipeloader is `use Pipeloader`, plus one flag for `auto_fuse`.
 
-```ruby
-Pipeloader.field_exact = true            # globally, before your types load, or…
+Run it: `ruby example/bench_wide.rb` (needs the seeded `graphql_experiment` DB).
 
-class Types::Post < GraphQL::Schema::Object
-  pipeloader_field_exact!                # …per type
-  field :title, String, null: false
-  field :author, Types::Author, null: false
+## Batch loaders for plain ActiveRecord
+
+The full surface behind the [quick-start above](#batch-loaders).
+
+```ruby
+class Author < ApplicationRecord
+  include Pipeloader::Batch::Model
+  batch_has_many   :books            # chainable, batched
+  batch_has_one    :profile
+  batch_belongs_to :publisher
 end
 ```
 
-For `{ posts { title author { name } } }` the posts SELECT becomes
-`SELECT id, title, author_id FROM …` (PK + selected column + the FK needed to
-resolve `author`), and the authors SELECT becomes `SELECT id, name FROM …` — for
-both the root relation and the `belongs_to`.
+`batch_has_many` / `batch_has_one` / `batch_belongs_to` declare a real AR association and
+accept everything the matching macro does (a scope, `class_name:`, `foreign_key:`, and so
+on). `batch_belongs_to` and `batch_has_one` return native records, batched the first time
+any sibling's target is read. `batch_has_many` returns a lazy, chainable proxy whose
+`where` / `order` / `limit` apply inside the one batched query (limit and offset are
+per-owner, top-N per group):
+
+```ruby
+authors.each { |a| a.books.where(published: true).to_a }   # filter pushed down, one query
+authors.each { |a| a.books.order(pages: :desc).limit(3) }  # each author's 3 longest, one query
+```
 
-**It never breaks a field.** A classifier narrows only when it can *prove* every
-selected field reads a known column or association. The instant a selection is
-opaque — a computed field, a custom resolver, anything it can't map to a column —
-it **bails to a whole-row fetch** for that record, so a projected field can never
-raise `MissingAttributeError`.
+The proxy covers the common read surface (`where`, `order`, `limit`, `select`, `pluck`,
+`find_by`, `exists?`, and `Enumerable`). Only writes (`<<`, `create`, `build`, ...)
+delegate to the real association; any read it doesn't implement raises `NoMethodError`
+rather than silently issuing a per-record query.
 
-**The `selects:` escape hatch.** A computed field can declare the columns it
-reads, so projection keeps them instead of bailing to a whole row:
+Counts and aggregates batch into a single `GROUP BY`:
 
 ```ruby
-field :excerpt, String, null: false, selects: %i[body]
-def excerpt = object.body[0, 200]
+batch_count     :books_count                                  # Integer, default 0
+batch_aggregate :total_pages, of: :books, function: :sum,     column: :pages
+batch_aggregate :longest,     of: :books, function: :maximum, column: :pages
+```
+
+For anything that isn't a plain association (an existence or viewer-scoped flag, a lookup
+by a non-PK column, a derived value) the general `batch` macro takes a loader returning a
+`{ key => value }` Hash, run once across all siblings:
+
+```ruby
+batch :viewer_has_starred, default: false do |book_ids|
+  Star.where(user_id: Current.user.id, book_id: book_ids).pluck(:book_id).index_with(true)
+end
 ```
 
-Selecting `excerpt` now adds `body` to the projection. With no opt-in (the
-default), `selects:` is accepted and ignored, and every SELECT is whole-row.
-
-## Status & caveats — this is a proof of concept
-
-- **Whole rows by default; field-exact is [opt-in](#field-exact-projection-opt-in).**
-  Off, AR picks the columns (maximum transparency); on, the pipeline projects to
-  the selected columns and bails to whole rows on anything opaque.
-- **PostgreSQL pipelines; SQLite narrows only; anything else raises.** Pipelining
-  is libpq-specific, so on PostgreSQL queries are pipelined, on SQLite they run
-  un-pipelined (the opt-in column projection still applies, useful for tests/dev),
-  and any other adapter raises a `RuntimeError` at query time rather than silently
-  misbehaving. Running SQLite un-pipelined is safe because SQLite is *embedded* —
-  its queries are in-process calls with no network round trip, so there's nothing
-  for a dataloader or a pipeline to collapse. N+1 there is just a series of cheap
-  local calls, not the latency amplification that makes N+1 catastrophic against a
-  networked database. So pipelining buys nothing on SQLite, and skipping it costs
-  nothing.
-- **Reads only.** It intercepts `select_all` (SELECTs); writes and non-SELECTs
-  fall straight through, and queries inside an open transaction are skipped.
-- **Assumes thread-isolated connections** (the ActiveRecord default): a request's
-  resolver fibers all share one connection. Under `:fiber` isolation you'd stash
-  per leased connection.
-- **Stats are process-global** single-threaded demo instrumentation.
-- Prepares and caches statements per connection, but doesn't re-prepare after a
-  reconnect / `DEALLOCATE` the way AR does. Also not hardened for multiple
-  databases, `count`/`exists?` (which route through other methods), or error
-  recovery mid-pipeline.
+[DATALOADERS.md](DATALOADERS.md) puts the common `GraphQL::Dataloader` sources
+(record-by-id, has-many, count, by-column, existence, derived) side by side with their
+batch-loader equivalents.
+
+Siblings are the records loaded by the same query, and the group rides on the records, so
+batching needs no setup and is correct across threads, fibers, `GraphQL::Dataloader`, and
+fiber-per-request servers alike. Records loaded on their own (a `find`, a separate query)
+form their own group and don't cross-batch. `has_many` / `has_one` / `belongs_to` (including
+`:through` and polymorphic), counts, and aggregates are covered; the `has_many` proxy is
+read-only.
+
+## Status and caveats
+
+A proof of concept.
+
+- Whole rows by default; [field-exact](#field-exact-projection-opt-in) is opt-in. Off, AR
+  picks the columns; on, the pipeline projects to the selected columns and falls back to
+  whole rows on anything opaque.
+- PostgreSQL pipelines, SQLite narrows only, anything else raises. Pipelining is
+  libpq-specific. On SQLite, queries run un-pipelined (the opt-in projection still
+  applies), which is safe because SQLite is embedded: its in-process queries have no round
+  trip to collapse, and N+1 there is just cheap local calls. Any other adapter raises a
+  `RuntimeError` at query time rather than misbehaving silently.
+- Reads only. It intercepts `select_all` (SELECTs); writes and non-SELECTs pass through,
+  and queries inside an open transaction are skipped.
+- Assumes thread-isolated connections (the ActiveRecord default): a request's resolver
+  fibers share one connection. Under `:fiber` isolation you'd stash per leased connection.
+- Stats are process-global, single-threaded demo instrumentation.
+- Statements are prepared once per request and `DEALLOCATE`d by the next one (piggybacked
+  onto its first burst, so cleanup adds no round trip), so no cache goes stale across a
+  reconnect or migration. A query error is drained and raised, leaving the connection
+  usable. Not yet hardened for multiple databases or `count`/`exists?` (which route through
+  other methods).
 
 ## Running the example
 
@@ -195,30 +265,42 @@ default), `selects:` is accepted and ignored, and every SELECT is whole-row.
 # Needs a Postgres DB with posts/authors/comments/users tables. In this repo:
 #   go run ./cmd/gqlbench -reset    # seeds the graphql_experiment DB
 ruby example/run.rb        # shows the round-trip collapse
-ruby example/bench.rb      # the latency benchmark (narrow tree)
-ruby example/bench_wide.rb # the wide-tree benchmark
+ruby example/bench_wide.rb # the latency benchmark
 ```
 
 Requires `activerecord`, `graphql`, and `pg` (libpq ≥ 14 for pipelining).
 
 ## Tests
 
-`rake test`. Three suites, all **parity-first** — the pipelined result must be
-byte-identical to plain ActiveRecord:
-
-- **`test/pipeloader_test.rb`** — every query runs through both a plain schema and
-  a `use Pipeloader` schema, asserting identical results across each relationship
-  kind, nullable foreign keys, empty has-many, deduplication, ordering, type
-  casting, aliases, variables, and multiplex. It also checks round-trip counts
-  (= tree depth) and that the patch leaves writes, transactions, and non-GraphQL
-  ActiveRecord untouched.
-- **`test/field_exact_test.rb`** — the opt-in projection: projected results match
-  the whole-row schema, the emitted SQL is actually narrowed (and keeps the FK),
-  the `selects:` escape hatch includes its columns, and opaque fields bail to a
-  whole-row `SELECT *` instead of raising.
-- **`test/adapter_test.rb`** — adapter handling: PostgreSQL pipelines, an
-  unsupported adapter raises, and a real in-memory **SQLite** run (in a subprocess)
-  proves projection works there with pipelining disabled.
-
-Needs a reachable Postgres (the suites create `pl_*` fixture tables in
-`graphql_experiment`).
+`rake test`. The pipelining suites are parity-first: the pipelined result must be
+byte-identical to plain ActiveRecord. The batch suites assert one query per level.
+
+- `test/pipeloader_test.rb`: every query runs through a plain schema and a `use Pipeloader`
+  schema, asserting identical results across each relationship kind, nullable foreign keys,
+  empty has-many, deduplication, ordering, type casting, aliases, variables, and multiplex.
+  It also checks round-trip counts (= tree depth), that the patch leaves writes,
+  transactions, and non-GraphQL ActiveRecord untouched, that a database error inside a
+  burst surfaces and leaves the connection usable, that prepared statements don't linger,
+  and that existing `GraphQL::Dataloader` sources keep working once pipeloader is installed.
+- `test/field_exact_test.rb`: projected results match the whole-row schema, the emitted SQL
+  is narrowed (and keeps the FK), the `selects:` hatch includes its columns, and opaque
+  fields fall back to a whole-row `SELECT *`.
+- `test/auto_fuse_test.rb`: a fused result is byte-identical to the un-fused path, fusion
+  collapses each level into one `ANY($1)` (round trips = depth, even on wide levels), and
+  every non-fusable shape falls back cleanly.
+- `test/adapter_test.rb`: PostgreSQL pipelines, an unsupported adapter raises, and a real
+  in-memory SQLite run (in a subprocess) proves projection works there with pipelining off.
+- `test/batch_*_test.rb`: the [batch loaders](#batch-loaders-for-plain-activerecord),
+  exhaustively. `batch_proxy_test` covers every `has_many`-proxy variant: `where` (hash,
+  string, range, `not`, chained, rewhere), `order` (asc, desc, multi-column, reorder, SQL
+  string), per-group `limit`/`offset`, `select`/`distinct`/`pluck`/`joins`, the
+  materializers, scope caching, and write-through. With it: `batch_singular_test`
+  (`belongs_to` including optional and a non-PK key, `has_one`), `batch_aggregate_test`
+  (count/sum/avg/min/max and defaults), `batch_through_test` (`:through` and polymorphic),
+  `batch_custom_test` (the `batch` macro), `batch_test` for the basics, and
+  `batch_context_test` for the sibling-group model (grouping by load, fiber- and
+  thread-safety by construction).
+
+Coverage: `rake coverage` (or `COVERAGE=1 rake test`) writes a SimpleCov report to
+`coverage/`. Needs a reachable Postgres (the suites create `pl_*` and `bl_*` fixture tables
+in `graphql_experiment`).

data/lib/pipeloader/ar_patch.rb

@@ -26,7 +26,9 @@ module Pipeloader
           sql, bind_objs, = to_sql_and_binds(relation, binds, preparable, allow_retry)
           if sql.lstrip[0, 6].casecmp("select").zero?
             params = bind_objs.map { |b| b.respond_to?(:value_for_database) ? b.value_for_database : b }
-            return dl.with(Pipeloader::Source, raw_connection).load([sql, params])
+            # Hand the Source this connection (self): it pipelines on raw_connection
+            # and types results via the adapter's OID lookup. One Source per checkout.
+            return dl.with(Pipeloader::Source, self).load([sql, params])
           end
         end
         # Synchronous fallback (no active response, or not gathered): one query,

data/lib/pipeloader/batch/batch_loader.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+module Pipeloader
+  module Batch
+    # Runs ONE query for a BatchProxy's accumulated scope across every live
+    # sibling of the owner, partitions rows by foreign key, applies per-group
+    # limit/offset, and caches each sibling's slice (keyed by the scope) so the
+    # other siblings' identical access is free.
+    module BatchLoader
+      module_function
+
+      def load(proxy)
+        owner = proxy.owner
+        cache_key = [proxy.name, proxy.cache_signature]
+        cache = owner._pipeloader_batch_scope_cache
+        return cache[cache_key] if cache.key?(cache_key)
+
+        siblings = relevant_siblings(proxy)
+        grouped = run(proxy, siblings)
+
+        siblings.each do |sibling|
+          sibling._pipeloader_batch_scope_cache[cache_key] = grouped[sibling.send(proxy.owner_key)]
+        end
+        grouped[owner.send(proxy.owner_key)]
+      end
+
+      def relevant_siblings(proxy)
+        owner = proxy.owner
+        siblings = owner._pipeloader_batch_context.all(owner.class).select(&:persisted?)
+        siblings << owner if owner.persisted? && siblings.none? { |sibling| sibling.equal?(owner) }
+        siblings
+      end
+
+      def run(proxy, siblings)
+        foreign_key = proxy.foreign_key
+        ids = siblings.map { |sibling| sibling.send(proxy.owner_key) }.compact.uniq
+
+        grouped = Hash.new { |hash, key| hash[key] = [] }
+        return grouped if ids.empty?
+
+        scope = proxy.relation.where(foreign_key => ids)
+        # A custom .select must still include the FK so we can partition the rows.
+        scope = scope.select(foreign_key) if scope.select_values.any?
+        scope.to_a.each { |record| grouped[record[foreign_key]] << record }
+
+        apply_window!(grouped, proxy) if proxy.limit_value || proxy.offset_value
+        grouped
+      end
+
+      # limit/offset are per group: the batched query is ordered but unlimited, and
+      # each owner's slice is taken in Ruby (so "5 per author", not 5 overall).
+      def apply_window!(grouped, proxy)
+        offset = proxy.offset_value || 0
+        limit = proxy.limit_value
+        grouped.each_key do |key|
+          windowed = grouped[key].drop(offset)
+          windowed = windowed.first(limit) if limit
+          grouped[key] = windowed
+        end
+      end
+    end
+  end
+end