emb 0.2.1 → 0.2.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 424790b9a9b835dd7febbdfa1df2e5f8b6a2a9b89290b5e653467fcc09e4f68f
-  data.tar.gz: d21f28bb4c79deaefa9bab91247f41ff5e84f869b422727939ab1f07a940eb34
+  metadata.gz: 2210ac3c666538429c3d0eccc58c26b94bc87a26f65cd6c3b4ba2f754b5de6fa
+  data.tar.gz: f576b3edf8bec7d38d156b35082e86ac19a80324c0f437e78197f674f0156215
 SHA512:
-  metadata.gz: eacbde1ca9ccf217bd53cb9028f8a5192427754abcee678373d38f16b08ce4ae00b5b905909cb0d7ef0f07fa04b39c0ad112c820c6f4fd6d3c98ba06708af854
-  data.tar.gz: 8605fe8eaede9567d6506f5247ef85dfcb32b9c1c1fa719d00ce2446233b0518cbd336e6f496c8449ac41ab33545776b05d1febe2bf3d5a576aafa47afa60d9c
+  metadata.gz: 3dc881cd39e7dcdf6898afa215d9d852a638a2df020d225056e861b9f79e57f13001a30c0e88b0b1cf0e997328c9682b6d4f2cdfbd97c7795e1adc6a8262a279
+  data.tar.gz: 93aa184f12078697b8f27946567dd3221b99d098f67f207d4cbe3842472583ae76fba8a8aa8bab5a5cba3a432c12eff2a2449d53ff93187a043a6bfd0e864c4d

data/README.md

@@ -44,12 +44,49 @@ Emb.setup
 2. `EMB_URL` environment variable
 3. Default: `redis://localhost:6379`
 
+### Global configuration
+
+Every client (`Emb.setup`, `Emb.new`, and the lazily-created default client) inherits a
+shared global `Emb::Config` value object set with `Emb.configure`. Settings
+resolve in this order — the first one wins:
+
+1. **Explicit per-call option** (`Emb.setup(pool: 10)`)
+2. **`Emb.configure` value**
+3. **Built-in default**
+
+```ruby
+Emb.configure do |c|
+  c.pool = 8
+  c.batch = false   # opt out of lazy batching app-wide
+end
+
+Emb.configuration   # => the shared Emb::Configuration
+Emb::Client.new(pool: 20)  # per-call still wins
+```
+
+`EMB_URL` remains the only environment variable (connection URL fallback, as before);
+`Emb.configure { |c| c.url = ... }` or an explicit `url:` override it.
+
+### Out-of-the-box defaults
+
+The shipped defaults are benchmark-derived (see `BENCHMARK.md`): **lazy batching is on by
+default** (`batch: true` — each embed coalesces into one `EMB.MULTI`), pool `5`, pure-Ruby
+RESP driver, `protocol: 2`, `reconnect_attempts: 3`. To keep the eager behavior (immediate
+`EMB` per call), opt out globally via `Emb.configure { |c| c.batch = false }` or per client
+with `Emb.new(batch: false)`.
+
 ### Connection pool
 
 ```ruby
 Emb.setup(url: "redis://localhost:6379", pool: 10)
 ```
 
+The default pool size is **5**. The pool is usually not the bottleneck for
+inference-bound workloads (small pools are fine); it becomes a knob only at high
+concurrency on a multi-model box — see [Performance](#performance). If a pool checkout
+would wait too long, `RedisClient`'s `connect_timeout`/`read_timeout` (above) bound the
+wait.
+
 ### Authentication
 
 If the server is configured with a password, include it in the URL:
@@ -89,8 +126,8 @@ Emb.setup(
 ```
 
 See the [redis-client documentation](https://github.com/redis-rb/redis-client) for
-all available options. Only `pool` is handled by the gem — everything else passes
-through to `RedisClient.new`.
+all available options. Only `pool` and `batch` are handled by the gem — everything
+else passes through to `RedisClient.new`.
 
 ## Instance-based clients
 
@@ -177,6 +214,144 @@ client.multi do |m|
 end
 ```
 
+### Lazy batching (`Emb.batch`)
+
+Instead of collecting pairs by hand, `Emb.batch` returns lazy embeddings that all
+coalesce into a single `EMB.MULTI` round trip when the first one is used. This is
+powered by the [batch-loader](https://github.com/exAspArk/batch-loader) gem.
+
+```ruby
+users = User.all # some application objects
+
+# Create loaders first...
+l1 = Emb.batch[:minilm]["hello"]
+l2 = Emb.batch[:minilm]["world"]
+l3 = Emb.batch[:bge]["bonjour"]
+
+# ...then consume them. The first use sends ONE EMB.MULTI for all three.
+l1.sum  # => 12.345
+l2.sum  # => -0.678
+l3.sum  # => 3.141
+```
+
+Instance clients expose the same API:
+
+```ruby
+client.batch[:minilm]["hello"].sum
+```
+
+Each lazy value materializes to the same shape as the eager API: a single text
+yields an `Array<Float>`, multiple texts yield `Array<Array<Float>>`:
+
+```ruby
+vec   = Emb.batch[:minilm]["hello"]   # use -> Array of Float
+vecs  = Emb.batch[:minilm]["hello", "world"]  # use -> Array of Array of Float
+```
+
+Embeddings are cached per thread, so reusing a lazy value (or creating an
+identical pair again in the same scope) is free after the first use. A pair whose
+embedding fails materializes as `nil`, matching `EMB.MULTI`'s per-pair null
+behavior; siblings in the same batch still succeed.
+
+#### The create-then-consume contract
+
+Loaders only fire when a value is **used**. Create all loaders *first*, then
+consume them, so they share one round trip:
+
+```ruby
+texts.each { |t| process(Emb.batch[:minilm][t]) }   # wrong: one MULTI per item
+loaders = texts.map { |t| Emb.batch[:minilm][t] }   # right: ONE MULTI for all
+loaders.each { |l| process(l) }
+```
+
+A loader that is created but never used **never embeds** (unless a sibling batch
+fires first) and is silently dropped when the thread's scope ends. Duration and
+scope: batching is per-thread — a multithreaded app issues one `EMB.MULTI` per
+thread per flush.
+
+### `batch` configuration option
+
+Setting `batch: true` makes the standard proxy API lazy, so existing call sites
+batch automatically without restructuring:
+
+```ruby
+Emb.setup(url: "redis://localhost:6379", batch: true)
+# or
+Emb.new(url: "redis://localhost:6379", batch: true)
+```
+
+With `batch: true`, `Emb[:minilm]["hello"]` returns a lazy embedding that sends
+`EMB.MULTI` on first use. The default is `false` — the proxy API stays eager,
+sending `EMB` immediately. `Emb.batch` works regardless of the option, and
+`Emb.multi` remains the explicit, eager, deterministic batching API.
+
+### Clearing the cache per request
+
+The per-thread batch scope holds cached embeddings for the life of the thread. In
+request-shaped processes (Rails, Rack apps, Sidekiq) mount `Emb::Middleware` to
+clear the scope at the end of each request:
+
+```ruby
+# config/application.rb (Rails)
+config.middleware.use Emb::Middleware
+
+# Any Rack app
+use Emb::Middleware
+```
+
+The scope is cleared even when the app raises, and a fresh scope starts
+automatically with the next request. Loaders created but never used within a
+request are dropped — the create-then-consume contract applies per request.
+
+## Performance
+
+### Eager-burst pipelining (no new API)
+
+For a burst of independent eager calls (feelers across call sites, fan-ins where
+batching doesn't fit), coalesce them into one packet with `RedisClient#pipelined`:
+
+```ruby
+client = Emb.new(url: "redis://localhost:6379")
+
+a = client.pool.with do |conn|
+  conn.pipelined do |pipe|
+    texts.each { |t| pipe.call("EMB", "minilm", t) }
+  end
+end
+# a -> Array of Float32-binary replies; unpack each with r.unpack("e*")
+```
+
+`Client#pool` exposes the pool's `RedisClient` connections. This measurably beats
+plain per-call round trips; use it for bursts, and `Emb.batch`/`Emb.multi` when you want
+server-side coalescing into one `EMB.MULTI`.
+
+### RESP driver: pure-Ruby default, `hiredis` on demand
+
+The pure-Ruby RESP parser is the default. The C `hiredis` driver only meaningfully helps
+the all-round-trip eager path (about +12% req/s) and is ~neutral for batched/pipelined
+workloads, so it is not worth the native-build dependency by default. Enable it when
+round-trip-heavy eager traffic dominates:
+
+```ruby
+require "hiredis-client"
+Emb.setup(url: "redis://localhost:6379", driver: :hiredis)
+```
+
+### Horizontal scaling
+
+emb instances are stateless — the model lives in memory and the LRU cache is per
+instance. Scale out without a cluster client:
+
+- **Model sharding** — run each instance with a subset of models; route by model via
+  dedicated clients (`Emb.new(url: "redis://model-a:6379")`).
+- **Text-keyed sharding** — several instances serving the same model behind an L4 /
+  load balancer for same-model scale.
+- **Cache warm-up** — because the cache is per instance, each new box starts cold
+  (optionally seed with `-cache` and a warmup pass).
+
+Lazy batching stays per server (a thread's loaders target one client), so batching and
+horizontal scale compose: each instance receives one `EMB.MULTI` per request.
+
 ### Commands
 
 ```ruby

data/lib/emb/batch.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+require 'batch_loader'
+
+module Emb
+  BATCH_KEY = :emb
+
+  BATCH_BLOCK = lambda do |items, loader, _args|
+    items.group_by(&:first).each do |client, client_items|
+      pairs = client_items.flat_map { |_, model, text| Array(text).flat_map { |t| [model.to_s, t] } }
+      results = Array(client.send_command('EMB.MULTI', *pairs))
+
+      offset = 0
+      client_items.each do |item|
+        _, _, text = item
+        texts = Array(text)
+        values = results[offset, texts.size].map { |entry| entry&.unpack('e*') }
+        offset += texts.size
+
+        # eager Proxy#[] shape: vector for a single text, vectors for many
+        loader.call(item, values.size == 1 ? values.first : values)
+      end
+    end
+  end
+
+  class << self
+    def build_batch_loader(client, model, text)
+      BatchLoader.for([client, model, text]).batch(key: BATCH_KEY, &BATCH_BLOCK)
+    end
+  end
+
+  class BatchProxy
+    def initialize(client)
+      @client = client
+      @models = {}
+    end
+
+    def [](name)
+      @models[name.to_sym] ||= BatchModelProxy.new(@client, name.to_sym)
+    end
+
+    def inspect
+      "#<Emb::BatchProxy client=#{@client.inspect}>"
+    end
+  end
+
+  class BatchModelProxy
+    attr_reader :name
+
+    def initialize(client, name)
+      @client = client
+      @name = name
+    end
+
+    def [](text, *texts)
+      Emb.build_batch_loader(@client, @name, texts.empty? ? text : [text, *texts])
+    end
+
+    def inspect
+      "#<Emb::BatchModelProxy #{@name}>"
+    end
+  end
+end

data/lib/emb/client.rb

@@ -4,19 +4,17 @@ require 'connection_pool'
 require 'redis_client'
 
 module Emb
-  DEFAULTS = { host: 'localhost', port: 6379, pool: 5 }.freeze
-
   class Client
     attr_reader :pool
 
-    def initialize(pool: DEFAULTS[:pool], **redis_options)
-      url = extract_url!(redis_options)
-      redis_options[:host] ||= DEFAULTS[:host] unless url
-      redis_options[:port] ||= DEFAULTS[:port] unless url
-      redis_options[:protocol] ||= 2
-      redis_options[:reconnect_attempts] ||= 3
+    def initialize(pool: nil, batch: nil, **redis_options)
+      cfg = Emb.configuration
+      @batch_enabled = batch.nil? ? cfg.batch : batch
+      size = pool.nil? ? cfg.pool : pool
+      url = extract_url!(redis_options, cfg)
+      redis_options = merged_redis_options(redis_options, cfg, url)
 
-      @pool = ConnectionPool.new(size: pool) do
+      @pool = ConnectionPool.new(size: size) do
         RedisClient.new(url: url, **redis_options)
       end
 
@@ -39,6 +37,14 @@ module Emb
       @registry[name] ||= Proxy.new(self, name.to_sym)
     end
 
+    def batch
+      @batch ||= BatchProxy.new(self)
+    end
+
+    def batch?
+      @batch_enabled
+    end
+
     def models
       raw = send_command('EMB.MODELS')
       return [] if raw.nil?
@@ -66,7 +72,7 @@ module Emb
     def ready
       send_command('EMB.READY')
 
-      "ready"
+      'ready'
     rescue RedisClient::CommandError => e
       e.message
     end
@@ -91,9 +97,23 @@ module Emb
 
     private
 
-    def extract_url!(opts)
+    def merged_redis_options(opts, cfg, url)
+      defaults = cfg.to_h
+      keys = defaults.keys - %i[url pool batch]
+      keys -= %i[host port] if url
+
+      keys.each do |key|
+        opts[key] = defaults[key] if opts[key].nil? && !defaults[key].nil?
+      end
+
+      opts
+    end
+
+    def extract_url!(opts, cfg)
       url = opts.delete(:url)
-      url.nil? ? ENV.fetch('EMB_URL', nil) : url
+      return url unless url.nil?
+
+      cfg.url || ENV.fetch('EMB_URL', nil)
     end
   end
 end

data/lib/emb/configuration.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+module Emb
+  class Configuration
+    OPTIONS = %i[
+      host port url pool batch driver protocol
+      connect_timeout read_timeout write_timeout reconnect_attempts
+    ].freeze
+
+    attr_accessor(*OPTIONS)
+
+    def initialize
+      self.host = 'localhost'
+      self.port = 6379
+      self.url = nil
+      self.pool = 5
+      self.batch = true
+      self.driver = nil
+      self.protocol = 2
+      self.connect_timeout = nil
+      self.read_timeout = nil
+      self.write_timeout = nil
+      self.reconnect_attempts = 3
+    end
+
+    def to_h
+      OPTIONS.to_h { |key| [key, public_send(key)] }
+    end
+  end
+end

data/lib/emb/middleware.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+require 'batch_loader'
+
+module Emb
+  # Rack middleware that clears the per-thread batch scope at the end of each
+  # request, bounding cache growth in long-lived request threads. The scope is
+  # cleared even when the app raises; a fresh scope starts with the next request.
+  class Middleware
+    def initialize(app)
+      @app = app
+    end
+
+    def call(env)
+      @app.call(env)
+    ensure
+      BatchLoader::Executor.clear_current
+    end
+  end
+end

data/lib/emb/proxy.rb

@@ -10,6 +10,8 @@ module Emb
     end
 
     def [](text, *texts)
+      return Emb.build_batch_loader(@client, @name, texts.empty? ? text : [text, *texts]) if @client.batch?
+
       set = Array(@client.send_command('EMB', @name.to_s, text, *texts))
       result = set.map { |entry| entry.unpack('e*') }
 

data/lib/emb.rb

@@ -1,9 +1,12 @@
 # frozen_string_literal: true
 
 require_relative 'emb/version'
+require_relative 'emb/configuration'
 require_relative 'emb/client'
 require_relative 'emb/proxy'
 require_relative 'emb/multi'
+require_relative 'emb/batch'
+require_relative 'emb/middleware'
 
 module Emb
   class << self
@@ -15,7 +18,17 @@ module Emb
 
     alias config setup
 
+    def configuration
+      @configuration ||= Configuration.new
+    end
+
+    def configure
+      yield configuration if block_given?
+      configuration
+    end
+
     def [](name)    = default_client[name]
+    def batch       = default_client.batch
     def models      = default_client.models
     def info(name)  = default_client.info(name)
     def stats       = default_client.stats

metadata

@@ -1,15 +1,29 @@
 --- !ruby/object:Gem::Specification
 name: emb
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.2.3
 platform: ruby
 authors:
 - elcuervo
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-07-02 00:00:00.000000000 Z
+date: 2026-08-24 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: batch-loader
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
 - !ruby/object:Gem::Dependency
   name: connection_pool
   requirement: !ruby/object:Gem::Requirement
@@ -48,7 +62,10 @@ files:
 - Gemfile
 - README.md
 - lib/emb.rb
+- lib/emb/batch.rb
 - lib/emb/client.rb
+- lib/emb/configuration.rb
+- lib/emb/middleware.rb
 - lib/emb/multi.rb
 - lib/emb/proxy.rb
 - lib/emb/version.rb