emb 0.2.1 → 0.2.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 424790b9a9b835dd7febbdfa1df2e5f8b6a2a9b89290b5e653467fcc09e4f68f
-  data.tar.gz: d21f28bb4c79deaefa9bab91247f41ff5e84f869b422727939ab1f07a940eb34
+  metadata.gz: ff0fe54e3a671a1df06f90f6da9317e1173f5df811bb5c95d4374de4487fd8fb
+  data.tar.gz: 9f9bfbde342b771c876f6ef8327b672a85c17c82c889f7ca35d3d4bdbe238cf8
 SHA512:
-  metadata.gz: eacbde1ca9ccf217bd53cb9028f8a5192427754abcee678373d38f16b08ce4ae00b5b905909cb0d7ef0f07fa04b39c0ad112c820c6f4fd6d3c98ba06708af854
-  data.tar.gz: 8605fe8eaede9567d6506f5247ef85dfcb32b9c1c1fa719d00ce2446233b0518cbd336e6f496c8449ac41ab33545776b05d1febe2bf3d5a576aafa47afa60d9c
+  metadata.gz: 0dc5d526c1e0314d7a41c9455fb9129c15f63a61d293eadde62aeb9c9b3f79aa40bad32351564e7b69ec9baf9524d74669129f3f46db8abc7d9eaccb7137145c
+  data.tar.gz: ac676d326318efc040277f7adbd0b8b064d2e563dc2f2516dbe6d6bccb96f9e468a35b0aed937b351af7e53cd87ff63128b5660d90a566201e5b665df5aba600

data/README.md

@@ -50,6 +50,12 @@ Emb.setup
 Emb.setup(url: "redis://localhost:6379", pool: 10)
 ```
 
+The default pool size is **5**. The pool is usually not the bottleneck for
+inference-bound workloads (small pools are fine); it becomes a knob only at high
+concurrency on a multi-model box — see [Performance](#performance). If a pool checkout
+would wait too long, `RedisClient`'s `connect_timeout`/`read_timeout` (above) bound the
+wait.
+
 ### Authentication
 
 If the server is configured with a password, include it in the URL:
@@ -89,8 +95,8 @@ Emb.setup(
 ```
 
 See the [redis-client documentation](https://github.com/redis-rb/redis-client) for
-all available options. Only `pool` is handled by the gem — everything else passes
-through to `RedisClient.new`.
+all available options. Only `pool` and `batch` are handled by the gem — everything
+else passes through to `RedisClient.new`.
 
 ## Instance-based clients
 
@@ -177,6 +183,144 @@ client.multi do |m|
 end
 ```
 
+### Lazy batching (`Emb.batch`)
+
+Instead of collecting pairs by hand, `Emb.batch` returns lazy embeddings that all
+coalesce into a single `EMB.MULTI` round trip when the first one is used. This is
+powered by the [batch-loader](https://github.com/exAspArk/batch-loader) gem.
+
+```ruby
+users = User.all # some application objects
+
+# Create loaders first...
+l1 = Emb.batch[:minilm]["hello"]
+l2 = Emb.batch[:minilm]["world"]
+l3 = Emb.batch[:bge]["bonjour"]
+
+# ...then consume them. The first use sends ONE EMB.MULTI for all three.
+l1.sum  # => 12.345
+l2.sum  # => -0.678
+l3.sum  # => 3.141
+```
+
+Instance clients expose the same API:
+
+```ruby
+client.batch[:minilm]["hello"].sum
+```
+
+Each lazy value materializes to the same shape as the eager API: a single text
+yields an `Array<Float>`, multiple texts yield `Array<Array<Float>>`:
+
+```ruby
+vec   = Emb.batch[:minilm]["hello"]   # use -> Array of Float
+vecs  = Emb.batch[:minilm]["hello", "world"]  # use -> Array of Array of Float
+```
+
+Embeddings are cached per thread, so reusing a lazy value (or creating an
+identical pair again in the same scope) is free after the first use. A pair whose
+embedding fails materializes as `nil`, matching `EMB.MULTI`'s per-pair null
+behavior; siblings in the same batch still succeed.
+
+#### The create-then-consume contract
+
+Loaders only fire when a value is **used**. Create all loaders *first*, then
+consume them, so they share one round trip:
+
+```ruby
+texts.each { |t| process(Emb.batch[:minilm][t]) }   # wrong: one MULTI per item
+loaders = texts.map { |t| Emb.batch[:minilm][t] }   # right: ONE MULTI for all
+loaders.each { |l| process(l) }
+```
+
+A loader that is created but never used **never embeds** (unless a sibling batch
+fires first) and is silently dropped when the thread's scope ends. Duration and
+scope: batching is per-thread — a multithreaded app issues one `EMB.MULTI` per
+thread per flush.
+
+### `batch` configuration option
+
+Setting `batch: true` makes the standard proxy API lazy, so existing call sites
+batch automatically without restructuring:
+
+```ruby
+Emb.setup(url: "redis://localhost:6379", batch: true)
+# or
+Emb.new(url: "redis://localhost:6379", batch: true)
+```
+
+With `batch: true`, `Emb[:minilm]["hello"]` returns a lazy embedding that sends
+`EMB.MULTI` on first use. The default is `false` — the proxy API stays eager,
+sending `EMB` immediately. `Emb.batch` works regardless of the option, and
+`Emb.multi` remains the explicit, eager, deterministic batching API.
+
+### Clearing the cache per request
+
+The per-thread batch scope holds cached embeddings for the life of the thread. In
+request-shaped processes (Rails, Rack apps, Sidekiq) mount `Emb::Middleware` to
+clear the scope at the end of each request:
+
+```ruby
+# config/application.rb (Rails)
+config.middleware.use Emb::Middleware
+
+# Any Rack app
+use Emb::Middleware
+```
+
+The scope is cleared even when the app raises, and a fresh scope starts
+automatically with the next request. Loaders created but never used within a
+request are dropped — the create-then-consume contract applies per request.
+
+## Performance
+
+### Eager-burst pipelining (no new API)
+
+For a burst of independent eager calls (feelers across call sites, fan-ins where
+batching doesn't fit), coalesce them into one packet with `RedisClient#pipelined`:
+
+```ruby
+client = Emb.new(url: "redis://localhost:6379")
+
+a = client.pool.with do |conn|
+  conn.pipelined do |pipe|
+    texts.each { |t| pipe.call("EMB", "minilm", t) }
+  end
+end
+# a -> Array of Float32-binary replies; unpack each with r.unpack("e*")
+```
+
+`Client#pool` exposes the pool's `RedisClient` connections. This measurably beats
+plain per-call round trips; use it for bursts, and `Emb.batch`/`Emb.multi` when you want
+server-side coalescing into one `EMB.MULTI`.
+
+### RESP driver: pure-Ruby default, `hiredis` on demand
+
+The pure-Ruby RESP parser is the default. The C `hiredis` driver only meaningfully helps
+the all-round-trip eager path (about +12% req/s) and is ~neutral for batched/pipelined
+workloads, so it is not worth the native-build dependency by default. Enable it when
+round-trip-heavy eager traffic dominates:
+
+```ruby
+require "hiredis-client"
+Emb.setup(url: "redis://localhost:6379", driver: :hiredis)
+```
+
+### Horizontal scaling
+
+emb instances are stateless — the model lives in memory and the LRU cache is per
+instance. Scale out without a cluster client:
+
+- **Model sharding** — run each instance with a subset of models; route by model via
+  dedicated clients (`Emb.new(url: "redis://model-a:6379")`).
+- **Text-keyed sharding** — several instances serving the same model behind an L4 /
+  load balancer for same-model scale.
+- **Cache warm-up** — because the cache is per instance, each new box starts cold
+  (optionally seed with `-cache` and a warmup pass).
+
+Lazy batching stays per server (a thread's loaders target one client), so batching and
+horizontal scale compose: each instance receives one `EMB.MULTI` per request.
+
 ### Commands
 
 ```ruby

data/lib/emb/batch.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+require 'batch_loader'
+
+module Emb
+  BATCH_KEY = :emb
+
+  BATCH_BLOCK = lambda do |items, loader, _args|
+    items.group_by(&:first).each do |client, client_items|
+      pairs = client_items.flat_map { |_, model, text| Array(text).flat_map { |t| [model.to_s, t] } }
+      results = Array(client.send_command('EMB.MULTI', *pairs))
+
+      offset = 0
+      client_items.each do |item|
+        _, _, text = item
+        texts = Array(text)
+        values = results[offset, texts.size].map { |entry| entry&.unpack('e*') }
+        offset += texts.size
+
+        # eager Proxy#[] shape: vector for a single text, vectors for many
+        loader.call(item, values.size == 1 ? values.first : values)
+      end
+    end
+  end
+
+  class << self
+    def build_batch_loader(client, model, text)
+      BatchLoader.for([client, model, text]).batch(key: BATCH_KEY, &BATCH_BLOCK)
+    end
+  end
+
+  class BatchProxy
+    def initialize(client)
+      @client = client
+      @models = {}
+    end
+
+    def [](name)
+      @models[name.to_sym] ||= BatchModelProxy.new(@client, name.to_sym)
+    end
+
+    def inspect
+      "#<Emb::BatchProxy client=#{@client.inspect}>"
+    end
+  end
+
+  class BatchModelProxy
+    attr_reader :name
+
+    def initialize(client, name)
+      @client = client
+      @name = name
+    end
+
+    def [](text, *texts)
+      Emb.build_batch_loader(@client, @name, texts.empty? ? text : [text, *texts])
+    end
+
+    def inspect
+      "#<Emb::BatchModelProxy #{@name}>"
+    end
+  end
+end

data/lib/emb/client.rb

@@ -9,7 +9,9 @@ module Emb
   class Client
     attr_reader :pool
 
-    def initialize(pool: DEFAULTS[:pool], **redis_options)
+    def initialize(pool: DEFAULTS[:pool], batch: false, **redis_options)
+      @batch_enabled = batch
+
       url = extract_url!(redis_options)
       redis_options[:host] ||= DEFAULTS[:host] unless url
       redis_options[:port] ||= DEFAULTS[:port] unless url
@@ -39,6 +41,14 @@ module Emb
       @registry[name] ||= Proxy.new(self, name.to_sym)
     end
 
+    def batch
+      @batch ||= BatchProxy.new(self)
+    end
+
+    def batch?
+      @batch_enabled
+    end
+
     def models
       raw = send_command('EMB.MODELS')
       return [] if raw.nil?
@@ -66,7 +76,7 @@ module Emb
     def ready
       send_command('EMB.READY')
 
-      "ready"
+      'ready'
     rescue RedisClient::CommandError => e
       e.message
     end

data/lib/emb/middleware.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+require 'batch_loader'
+
+module Emb
+  # Rack middleware that clears the per-thread batch scope at the end of each
+  # request, bounding cache growth in long-lived request threads. The scope is
+  # cleared even when the app raises; a fresh scope starts with the next request.
+  class Middleware
+    def initialize(app)
+      @app = app
+    end
+
+    def call(env)
+      @app.call(env)
+    ensure
+      BatchLoader::Executor.clear_current
+    end
+  end
+end

data/lib/emb/proxy.rb

@@ -10,6 +10,8 @@ module Emb
     end
 
     def [](text, *texts)
+      return Emb.build_batch_loader(@client, @name, texts.empty? ? text : [text, *texts]) if @client.batch?
+
       set = Array(@client.send_command('EMB', @name.to_s, text, *texts))
       result = set.map { |entry| entry.unpack('e*') }
 

data/lib/emb.rb

@@ -4,6 +4,8 @@ require_relative 'emb/version'
 require_relative 'emb/client'
 require_relative 'emb/proxy'
 require_relative 'emb/multi'
+require_relative 'emb/batch'
+require_relative 'emb/middleware'
 
 module Emb
   class << self
@@ -16,6 +18,7 @@ module Emb
     alias config setup
 
     def [](name)    = default_client[name]
+    def batch       = default_client.batch
     def models      = default_client.models
     def info(name)  = default_client.info(name)
     def stats       = default_client.stats

metadata

@@ -1,15 +1,29 @@
 --- !ruby/object:Gem::Specification
 name: emb
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.2.2
 platform: ruby
 authors:
 - elcuervo
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-07-02 00:00:00.000000000 Z
+date: 2026-08-24 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: batch-loader
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
 - !ruby/object:Gem::Dependency
   name: connection_pool
   requirement: !ruby/object:Gem::Requirement
@@ -48,7 +62,9 @@ files:
 - Gemfile
 - README.md
 - lib/emb.rb
+- lib/emb/batch.rb
 - lib/emb/client.rb
+- lib/emb/middleware.rb
 - lib/emb/multi.rb
 - lib/emb/proxy.rb
 - lib/emb/version.rb