ndb 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: d05135b99146700c912046914e228098dc99c4bd67181074f39ea399c1d2e375
+  data.tar.gz: 76be9b065c8505b1a7c59ce69bf66bce0aa179bb6e4368f73facaea10e30c848
+SHA512:
+  metadata.gz: b3e6fa3bded8d9ee8824f7c0038e2fe74c462cdd95561b20f0a1548c59b40ff825bb1cd41348c9a2b919166fba3ba2ded2c5b58f61133cd729adf2f915e7c0b9
+  data.tar.gz: 5ace970046bad037e2977795771133017dc594ef83f64ee914799d732fb4b71baca7446e2788464644d9e56e75320d0c7987f87305801730a36a1f043d4a6ceb

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 thadeu
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,296 @@
+# ndb
+
+<img src="https://raw.githubusercontent.com/thadeu/ndb/main/brand/png/lockup-on-light.png" width="220">
+
+Plain Ruby client for [ndb](https://github.com/thadeu/ndb), an NDJSON query daemon. It talks HTTP over a unix socket or TCP, with a connection pool and keep-alive. It uses the Ruby standard library only: **no runtime dependencies**. The gem only builds JSON. The daemon decides what every query means.
+
+Building a Rails app with models? Use [`ndb-rails`](https://github.com/thadeu/ndb/tree/main/sdk/rails) on top of this gem.
+
+Docs: [client guide](https://github.com/thadeu/ndb/blob/main/docs/dev/clients.md), [query language](https://github.com/thadeu/ndb/blob/main/docs/dev/query.md), [HTTP API](https://github.com/thadeu/ndb/blob/main/docs/dev/http-api.md).
+
+## Install
+
+Needs Ruby 3.2 or newer and a running ndb daemon.
+
+```ruby
+# Gemfile
+gem "ndb"
+```
+
+```sh
+bundle add ndb
+# or: gem install ndb
+```
+
+```ruby
+require "ndb"
+```
+
+## Connect
+
+```ruby
+ndb = Ndb::Client.new(socket: "/run/ndb/ndb.sock")                       # unix socket
+ndb = Ndb::Client.new(url: "unix:///run/ndb/ndb.sock")                   # same, as a URL
+ndb = Ndb::Client.new(url: "tcp://10.0.0.5:7070", token: "secret")       # TCP with a bearer token
+ndb = Ndb::Client.new                                                    # reads NDB_URL and NDB_TOKEN
+```
+
+All options:
+
+```ruby
+Ndb::Client.new(
+  url: nil,            # "unix:///path", "tcp://host:port", or a bare path that starts with /
+  socket: nil,         # a unix socket path
+  token: nil,          # bearer token
+  timeout: 30,         # seconds, per request
+  pool_size: 16,       # idle connections kept
+  ready_timeout: 10    # seconds to retry while the daemon starts; 0 turns it off
+)
+```
+
+- The address is, in order: `url:`, `socket:`, env `NDB_URL`, then the default socket `/var/run/ndb/ndb.sock`.
+- The token is `token:`, or env `NDB_TOKEN`. It goes only in the `Authorization` header. A unix socket does not need it.
+- A bad address raises `ArgumentError` at once.
+- `ready_timeout` is how long a call retries while the daemon is not ready (`not_ready`, or the socket is missing or refuses). It uses exponential backoff (50 ms up to 1 s) with jitter. A write is retried only when the daemon certainly did not process it. A read is also retried after a dropped connection. A timeout is never retried. When the deadline passes, the last error is raised.
+- A reused keep-alive connection that the server closed is sent again only for a read. A write is never sent twice.
+- The pool is thread-safe and drops its connections after a `fork`.
+
+## Collections
+
+```ruby
+ndb.ensure_collection("calls",
+                      id: "call_id",
+                      indexes: ["status"],
+                      unique: ["caller"],
+                      time: "at")      # also: expires_field:, ttl:
+
+ndb.collections   # => [{"name" => "calls", ...}]
+```
+
+`ensure_collection` is idempotent. A different definition for an existing collection raises `Ndb::CatalogConflict`.
+
+## Insert, update, delete
+
+Every document has a version. A new document has version 1, and each change adds 1.
+
+```ruby
+results = ndb.insert("calls", [
+  { call_id: "c1", status: "answered", caller: "a", duration: 42 },
+  { call_id: "c2", status: "failed", caller: "b", duration: 5 }
+])
+results   # => [{"id" => "c1", "v" => 1}, {"id" => "c2", "v" => 1}]
+
+ndb.update("calls", "c1", set: { status: "done" }, unset: ["caller"])   # => {"id" => "c1", "v" => 2}
+ndb.update("calls", "c1", set: { duration: 50 }, if_version: 2)         # only if the version is still 2
+
+ndb.delete("calls", "c2")
+ndb.delete("calls", "c1", if_version: 3)
+```
+
+- `if_version:` is optimistic locking. If another writer changed the document first, you get `Ndb::VersionConflict` (code `version_conflict`). Reload and try again.
+- A duplicate id raises `Ndb::UniqueViolation` (code `duplicate_id`). A broken unique field raises it too (code `unique_violation`).
+- A missing document raises `Ndb::NotFound`.
+
+```ruby
+begin
+  ndb.update("calls", "c1", set: { status: "done" }, if_version: version)
+rescue Ndb::VersionConflict
+  version = ndb.find("calls", where: { call_id: "c1" }, versions: true).rows.first["_v"]
+  retry
+end
+```
+
+## Find
+
+`find` returns a `Ndb::Page` with `rows` (an array of Hash), `next_cursor`, `plan`, `examined` and `total`.
+
+```ruby
+page = ndb.find("calls",
+                where: { status: "answered", duration: { "$gte" => 30, "$lte" => 60 } },
+                sort: { at: -1 },
+                fields: %w[call_id status],
+                limit: 50,
+                skip: 0)
+
+page.rows          # => [{"call_id" => "c1", "status" => "answered"}, ...]
+page.plan          # "id", "index:status", "scan" or "snapshot"
+page.examined      # rows the daemon looked at
+```
+
+The body of `find` is the JSON of the [query language](https://github.com/thadeu/ndb/blob/main/docs/dev/query.md): `where`, `sort`, `fields`, `limit`, `skip`, `after`, `count`, `explain`, `versions`. Symbol keys are fine.
+
+In `find`, `where` is sent as it is: write the daemon operators yourself. (Only `watch` turns Ruby values such as Range and Array into operators, see below.)
+
+```ruby
+ndb.find("calls", where: { status: { "$in" => %w[answered failed] } })
+ndb.find("calls", where: { duration: { "$gte" => 30, "$lte" => 60 } })
+ndb.find("calls", where: { "$or" => [{ status: "failed" }, { duration: { "$gt" => 50 } }] })
+ndb.find("calls", where: { "agent.id" => 7 })   # dot path
+```
+
+- `sort: { at: -1, call_id: 1 }`: `1` ascending, `-1` descending. The key order is the sort order. The id is always the last key.
+- `versions: true` adds the field `_v` (the document version) to each row.
+
+### Regular expressions
+
+A Ruby `Regexp` is accepted anywhere a value is: a field value, `"$regex" => /x/`, inside `$not`, `$or`, `$and`, in `find`, `aggregate` (`$match`) and in the `where:` of `watch`.
+
+```ruby
+ndb.find("calls", where: { name: /^alpha/i })                    # i: ignore case
+ndb.find("calls", where: { name: { "$regex" => /bet$/ } })
+ndb.find("calls", where: { "$or" => [{ name: /^a/ }, { name: /^b/ }] })
+```
+
+The daemon uses RE2. The mapping to `$options` is:
+
+| Ruby | `$options` |
+|---|---|
+| always | `m` (Ruby `^` and `$` match at each line, so the daemon must do the same) |
+| `/i` (IGNORECASE) | `i` |
+| `/m` (MULTILINE: `.` matches a newline) | `s` |
+| `/x` (EXTENDED) | `x` |
+
+Ruby features that RE2 does not have raise `Ndb::InvalidQuery` (code `invalid_query`) **before** any request: look-around (`(?=`, `(?!`, `(?<=`, `(?<!`), backreferences (`\1` to `\9`, `\k<name>`), atomic groups (`(?>`), possessive quantifiers (`a++`), `\h`, `\G`, `\R` and `\X`.
+
+A String pattern is sent unchanged and has no `m`: `where: { name: { "$regex" => "^abc" } }`. A pattern anchored with `^` and a literal start, without `i` or `m`, uses the index on the field (`page.plan` is `index:name`). A Regexp always sends `m`, so it is a scan filter, not an index range. For a prefix search that uses the index, send the String form.
+
+### Count
+
+```ruby
+page = ndb.find("calls", where: { status: "answered" }, count: true, limit: 1)
+page.total   # how many documents match, whatever the limit
+```
+
+### Cursor pagination
+
+```ruby
+cursor = nil
+
+loop do
+  page = ndb.find("calls", where: { status: "answered" }, sort: { call_id: 1 }, limit: 100, after: cursor)
+
+  page.rows.each { |row| puts row["call_id"] }
+
+  cursor = page.next_cursor
+
+  break if cursor.nil?   # nil on the last page
+end
+```
+
+A cursor works only with the same `where` and `sort`. Another query shape raises `Ndb::InvalidCursor`.
+
+## Aggregate
+
+The pipeline needs exactly one `$group`. Accumulators: `$count`, `$sum`, `$avg`, `$min`, `$max`, `$first`, `$last`.
+
+```ruby
+page = ndb.aggregate("calls",
+                     pipeline: [
+                       { "$match" => { "duration" => { "$gte" => 1 } } },
+                       { "$group" => {
+                         "_id" => "$status",
+                         "n" => { "$count" => {} },
+                         "avg" => { "$avg" => "$duration" }
+                       } },
+                       { "$sort" => { "n" => -1 } }
+                     ],
+                     limit: 100,
+                     count: true)
+
+page.rows    # => [{"_id" => "answered", "n" => 2, "avg" => 35.5}, ...]
+page.total   # with count: true
+```
+
+`$dateTrunc` units are `minute`, `hour`, `day`, `week` and `month`.
+
+## Watch
+
+A watch streams the changes of a collection to a named consumer: first the events after its last ack, then new events as they happen. Use it for outbox-style work.
+
+```ruby
+begin
+  ndb.watch("calls", "relay", where: { status: "failed" }) do |event|
+    # event has "pos", "op" ("put" or "del"), "id", "v" and "doc"
+    handle(event)
+
+    ndb.ack("calls", "relay", event["pos"])
+  end
+rescue Ndb::WatchLagged
+  retry # the consumer was too slow. watch continues after the last ack.
+rescue Ndb::HistoryTruncated
+  raise # the saved position is too old. Start over.
+end
+```
+
+- `where:` takes the same conditions as the models of `ndb-rails`: equality, an Array (`$in`), a Range, a Regexp (see Regular expressions), `nil` (`$exists: false`) and `$` operators as keys. Only matching events come. A delete event has no `"doc"`; it is sent only if this stream saw an earlier matching put of that id.
+- `from: "now"` starts a new consumer at the end. `limit: n` ends the stream after `n` events.
+- Without a block, `watch` returns an Enumerator: `ndb.watch("calls", "relay", limit: 10).to_a`.
+- Delivery is at least once. After a restart an event can come again, so make the work safe to repeat. `ack` stores the position of an event: the next `watch` with that consumer name starts after it.
+- `HistoryTruncated` means the WAL files for the position were deleted (`wal_retention`).
+- A watch holds one connection until it ends. It is retried only for the initial open, never in the middle of a stream.
+
+## Blobs
+
+Large values live in the blob store. A document keeps only a reference.
+
+```ruby
+stored = ndb.put_blob(File.open("call.wav"), "audio/wav")   # an IO, streamed in chunks
+stored   # => {"$blob" => "sha256:...", "size" => 123456, "type" => "audio/wav"}
+
+ndb.insert("calls", [{ call_id: "c9", recording: stored }])
+
+ndb.get_blob(stored["$blob"])                    # all the bytes
+ndb.get_blob(stored["$blob"], from: 0, to: 1023) # bytes 0..1023 (inclusive)
+```
+
+The same content always gives the same reference. An IO upload is never retried.
+
+## Errors
+
+All errors inherit from `Ndb::Error`, which has `code` (the daemon code) and `status` (the HTTP status).
+
+| Class | Code |
+|---|---|
+| `Unauthorized` | `unauthorized` |
+| `NotReady` | `not_ready` |
+| `NotFound` | `not_found` |
+| `UnknownCollection` | `unknown_collection` |
+| `VersionConflict` | `version_conflict` |
+| `UniqueViolation` | `unique_violation`, `duplicate_id` |
+| `CatalogConflict` | `catalog_conflict` |
+| `CatalogInvalid` | `catalog_invalid` |
+| `ReadOnlyCollection` | `direct_collection` |
+| `InvalidQuery` | `invalid_query`, `invalid_request` |
+| `InvalidCursor` | `cursor_invalid`, `cursor_mismatch`, `cursor_expired` |
+| `WatchLagged` | `watch_lagged` |
+| `HistoryTruncated` | `history_truncated` |
+| `ConnectionError` | cannot reach the daemon. Subclasses: `ConnectError` (nothing was sent), `TimeoutError` |
+
+Any other code is a plain `Ndb::Error`. A response that is not a daemon error has the code `http_status`.
+
+```ruby
+begin
+  ndb.insert("calls", [{ call_id: "c1" }])
+rescue Ndb::UniqueViolation => e
+  e.code    # "duplicate_id"
+  e.status  # 409
+end
+```
+
+## Testing
+
+The tests of this gem build the daemon from the repository (`go build ./cmd/ndb`), start it on a short temp unix socket (macOS allows 104 bytes), wait for `GET /health` to return 200, and stop it at exit. To do the same by hand:
+
+```sh
+ndb -config /nonexistent.yml -data-dir "$(mktemp -d)" -unix /tmp/ndb-test.sock
+```
+
+```ruby
+ndb = Ndb::Client.new(socket: "/tmp/ndb-test.sock")
+```
+
+Use a new collection name per test. Run the suite with `bundle exec rake test` in `sdk/ruby`.
+
+## License
+
+MIT.

data/lib/ndb/client.rb

@@ -0,0 +1,245 @@
+# frozen_string_literal: true
+
+require "json"
+require "uri"
+
+module Ndb
+  Page = Struct.new(:rows, :next_cursor, :plan, :examined, :total)
+
+  # The HTTP API of the daemon (plan §12.1). It builds JSON and reads
+  # NDJSON; it has no query semantics.
+  class Client
+    DEFAULT_SOCKET = "/var/run/ndb/ndb.sock"
+
+    attr_reader :pool, :endpoint
+
+    RETRY_BASE = 0.05
+    RETRY_CAP = 1.0
+
+    # The address is, in order: url:, socket:, env NDB_URL, the default
+    # socket. The token is token:, or env NDB_TOKEN. It goes only in the
+    # Authorization header, and the unix socket does not need it.
+    #
+    # ready_timeout is how many seconds a call retries while the daemon is
+    # starting or recovering (not_ready, socket missing or refusing), with
+    # exponential backoff and jitter. 0 turns it off.
+    def initialize(url: nil, socket: nil, token: nil, timeout: 30, pool_size: 16, ready_timeout: 10)
+      @endpoint = Endpoint.parse(url || socket || presence(ENV["NDB_URL"]) || DEFAULT_SOCKET)
+      @token = presence(token) || presence(ENV["NDB_TOKEN"])
+      @pool = Pool.new(@endpoint, timeout: timeout, size: pool_size)
+      @ready_timeout = ready_timeout.to_f
+    end
+
+    def socket = @endpoint.path
+
+    def find(coll, body)
+      page(post(coll, "find", Conditions.regexps(body)))
+    end
+
+    def aggregate(coll, body)
+      page(post(coll, "aggregate", Conditions.regexps(body)))
+    end
+
+    def insert(coll, docs)
+      json(post(coll, "insert", insert: docs)).fetch("results")
+    end
+
+    def update(coll, id, set: {}, unset: [], if_version: nil)
+      change = { id: id, set: set, unset: unset }
+      change[:if_version] = if_version if if_version
+
+      json(post(coll, "update", update: change))
+    end
+
+    def delete(coll, id, if_version: nil)
+      change = { id: id }
+      change[:if_version] = if_version if if_version
+
+      json(post(coll, "delete", delete: change))
+    end
+
+    # Streams the changes of a collection for a named consumer: first the ones
+    # after its last ack, then the new ones. Yields one Hash per event (pos,
+    # op, id, v, doc). where is the same Hash/Range conditions as find, and
+    # only the matching events come; limit ends the stream after that many
+    # events. Without a block it returns an Enumerator. Delivery is at least
+    # once: ack the pos of an event when its work is done.
+    #
+    # The stream raises WatchLagged when the consumer is too slow, and
+    # HistoryTruncated when its position is older than the kept WAL; call
+    # watch again for the first, start over for the second.
+    def watch(coll, consumer, where: nil, from: nil, limit: nil, &block)
+      return enum_for(:watch, coll, consumer, where: where, from: from, limit: limit) unless block
+
+      body = { consumer: consumer, from: from, limit: limit }.compact
+      body[:where] = Conditions.equality(where) if where
+
+      with_ready_retry { stream_events("#{path(coll)}/watch", body, &block) }
+    end
+
+    # Stores the position of an event for its consumer: the next watch with
+    # that consumer name starts after it.
+    def ack(coll, consumer, pos)
+      json(post(coll, "ack", consumer: consumer, pos: pos))
+    end
+
+    def ensure_collection(coll, definition)
+      json(request("PUT", path(coll), definition))
+    end
+
+    def collections
+      json(request("GET", "/v1/c", nil)).fetch("collections")
+    end
+
+    # Uploads an IO (streamed) and returns the reference hash.
+    def put_blob(io, content_type = nil)
+      json(request("PUT", "/v1/blobs", io, headers: { "Content-Type" => content_type || "application/octet-stream" }))
+    end
+
+    # The bytes of a blob, or of its range from..to (inclusive).
+    def get_blob(ref, from: nil, to: nil)
+      headers = from ? { "Range" => "bytes=#{from}-#{to}" } : {}
+
+      request("GET", "/v1/blobs/#{ref}", nil, headers: headers).body
+    end
+
+    private
+
+    def presence(value)
+      value.to_s.strip.empty? ? nil : value
+    end
+
+    def post(coll, verb, body)
+      request("POST", "#{path(coll)}/#{verb}", body)
+    end
+
+    def path(coll)
+      "/v1/c/#{URI.encode_www_form_component(coll).gsub('+', '%20')}"
+    end
+
+    def request(method, target, body, headers: {})
+      payload = encode(body)
+      headers = authorize(headers)
+
+      safe = idempotent?(method, target) && !payload.respond_to?(:read)
+
+      with_ready_retry(idempotent: safe) do
+        response = @pool.with { |conn| conn.request(method, target, payload, headers: headers, idempotent: safe) }
+
+        raise_api_error(response) if response.status >= 300
+
+        response
+      end
+    end
+
+    def idempotent?(method, target)
+      method == "GET" || target.match?(%r{/(find|count|aggregate|get|stats)\z})
+    end
+
+    # Runs the block again while the daemon is not ready. It repeats only
+    # when the request certainly was not processed (not_ready, a failed
+    # connect), or for a read when the connection dropped. Never a timeout.
+    def with_ready_retry(idempotent: false)
+      deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + @ready_timeout
+      delay = RETRY_BASE
+
+      begin
+        yield
+      rescue NotReady, ConnectError, ConnectionError => e
+        raise if @ready_timeout <= 0 || e.is_a?(TimeoutError)
+        raise unless e.is_a?(NotReady) || e.is_a?(ConnectError) || idempotent
+
+        wait = (delay / 2) + (rand * delay / 2)
+
+        raise if Process.clock_gettime(Process::CLOCK_MONOTONIC) + wait > deadline
+
+        sleep wait
+
+        delay = [delay * 2, RETRY_CAP].min
+
+        retry
+      end
+    end
+
+    def authorize(headers)
+      @token ? headers.merge("Authorization" => "Bearer #{@token}") : headers
+    end
+
+    # JSON for a Hash, "" for nothing, an IO as it is (blob upload).
+    def encode(body)
+      return "" if body.nil?
+      return body if body.respond_to?(:read)
+
+      JSON.generate(body)
+    end
+
+    def json(response)
+      JSON.parse(response.body)
+    end
+
+    def page(response)
+      headers = response.headers
+
+      Page.new(
+        ndjson(response.body),
+        headers["x-ndb-next"],
+        headers["x-ndb-plan"],
+        headers["x-ndb-examined"]&.to_i,
+        headers["x-ndb-total"]&.to_i
+      )
+    end
+
+    def ndjson(body)
+      body.each_line.filter_map do |line|
+        line = line.strip
+
+        JSON.parse(line) unless line.empty?
+      end
+    end
+
+    def stream_events(target, body, &block)
+      pending = +""
+
+      response = @pool.with do |conn|
+        conn.stream("POST", target, JSON.generate(body), headers: authorize({})) do |chunk|
+          pending << chunk
+
+          while (newline = pending.index("\n"))
+            emit_event(pending.slice!(0, newline + 1), &block)
+          end
+        end
+      end
+
+      raise_api_error(response) if response.status >= 300
+    end
+
+    # A heartbeat is skipped; an error line raises.
+    def emit_event(line)
+      line = line.strip
+
+      return if line.empty?
+
+      event = JSON.parse(line)
+
+      return if event["heartbeat"]
+
+      if (error = event["error"])
+        raise Ndb.error_for(error["code"], error["message"], 200)
+      end
+
+      yield event
+    end
+
+    def raise_api_error(response)
+      parsed = JSON.parse(response.body)
+
+      raise Ndb.error_for("not_ready", "ndb is recovering", 503) if response.status == 503 && parsed["status"] == "recovering"
+
+      error = parsed.fetch("error")
+
+      raise Ndb.error_for(error["code"], error["message"], response.status)
+    rescue JSON::ParserError, KeyError
+      raise Error.new("HTTP #{response.status}: #{response.body.strip}", code: "http_status", status: response.status)
+    end
+  end
+end

data/lib/ndb/conditions.rb

@@ -0,0 +1,184 @@
+# frozen_string_literal: true
+
+require "date"
+
+module Ndb
+  # Ruby values -> the where JSON of the daemon (plan §12.2). No semantics:
+  # every operator is evaluated by the daemon.
+  module Conditions
+    module_function
+
+    # {status: "x", n: 1..3} -> {"status" => "x", "n" => {"$gte" => 1, "$lte" => 3}}
+    # A key that starts with "$" ("$or", "$and", "$not") is passed as is.
+    def equality(attrs)
+      attrs.to_h do |field, value|
+        key = field.to_s
+
+        [key, key.start_with?("$") ? json_value(value) : condition(value)]
+      end
+    end
+
+    # where.not: every field is negated on its own.
+    def negation(attrs)
+      attrs.each_with_object({}) do |(field, value), out|
+        key = field.to_s
+
+        case value
+        when nil then out[key] = { "$exists" => true }
+        when Array then out[key] = { "$nin" => json_value(value) }
+        when Regexp then out[key] = { "$not" => json_value(value) }
+        when Range then out["$not"] = { key => condition(value) }
+        when Hash then out["$not"] = { key => condition(value) }
+        else out[key] = { "$ne" => json_value(value) }
+        end
+      end
+    end
+
+    # AND of clauses: flat when the keys are apart, "$and" when they collide.
+    def combine(clauses)
+      clauses.reduce({}) do |acc, clause|
+        next acc.merge(clause) if (acc.keys & clause.keys).empty?
+
+        { "$and" => [acc, clause] }
+      end
+    end
+
+    def condition(value)
+      case value
+      when nil then { "$exists" => false }
+      when Array then { "$in" => json_value(value) }
+      when Range then range(value)
+      when Hash then json_value(value)
+      else json_value(value)
+      end
+    end
+
+    def range(range)
+      out = {}
+
+      out["$gte"] = json_value(range.begin) unless range.begin.nil?
+
+      unless range.end.nil?
+        out[range.exclude_end? ? "$lt" : "$lte"] = json_value(range.end)
+      end
+
+      out
+    end
+
+    # Times become RFC 3339 strings with milliseconds; the rest is JSON as is.
+    def json_value(value)
+      case value
+      when Time, DateTime then value.utc.iso8601(3)
+      when Date then value.iso8601
+      when Symbol then value.to_s
+      when Array then value.map { |v| json_value(v) }
+      when Regexp then regex(value)
+      when Hash then hash_value(value)
+      else time_like?(value) ? value.utc.iso8601(3) : value
+      end
+    end
+
+    def hash_value(hash)
+      hash.each_with_object({}) do |(key, value), out|
+        if key.to_s == "$regex" && value.is_a?(Regexp)
+          out.merge!(regex(value))
+        else
+          out[key.to_s] = json_value(value)
+        end
+      end
+    end
+
+    # Only the Regexp objects of a request body become $regex; every other
+    # value is left as it is (find and aggregate send the body as it is).
+    def regexps(value)
+      case value
+      when Regexp then regex(value)
+      when Array then value.map { |v| regexps(v) }
+      when Hash
+        value.each_with_object({}) do |(key, v), out|
+          if key.to_s == "$regex" && v.is_a?(Regexp)
+            out.merge!(regex(v))
+          else
+            out[key] = regexps(v)
+          end
+        end
+      else value
+      end
+    end
+
+    # A Ruby Regexp -> {"$regex" => source, "$options" => flags}. The flag m
+    # is always set: in Ruby ^ and $ match at each line. IGNORECASE is i,
+    # MULTILINE (dot matches a newline) is s, EXTENDED is x.
+    def regex(regexp)
+      reject_unsupported!(regexp.source)
+
+      options = +"m"
+
+      options << "i" if regexp.options.anybits?(Regexp::IGNORECASE)
+      options << "s" if regexp.options.anybits?(Regexp::MULTILINE)
+      options << "x" if regexp.options.anybits?(Regexp::EXTENDED)
+
+      { "$regex" => regexp.source, "$options" => options.chars.sort_by { |c| "imsx".index(c) }.join }
+    end
+
+    # The daemon uses RE2. These Ruby features do not exist there, so they
+    # fail here, before any request.
+    def reject_unsupported!(source)
+      feature = unsupported_feature(source)
+
+      return unless feature
+
+      raise InvalidQuery.new("regexp /#{source}/ uses #{feature}, which RE2 does not support",
+                             code: "invalid_query")
+    end
+
+    LOOKAROUND = { "?=" => "look-ahead (?=", "?!" => "negative look-ahead (?!",
+                   "?<=" => "look-behind (?<=", "?<!" => "negative look-behind (?<!",
+                   "?>" => "an atomic group (?>" }.freeze
+    ESCAPES = { "h" => "\\h", "G" => "\\G", "R" => "\\R", "X" => "\\X", "k" => "\\k<name>" }.freeze
+
+    def unsupported_feature(source)
+      in_class = false
+      quantifier = false
+      i = 0
+
+      while i < source.length
+        char = source[i]
+
+        if char == "\\"
+          nxt = source[i + 1].to_s
+
+          return "a backreference \\#{nxt}" if nxt.match?(/[1-9]/)
+          return ESCAPES[nxt] if ESCAPES.key?(nxt)
+
+          quantifier = false
+          i += 2
+
+          next
+        end
+
+        if in_class
+          in_class = false if char == "]"
+        elsif char == "["
+          in_class = true
+          i += 1 if source[i + 1] == "^"
+          i += 1 if source[i + 1] == "]"
+        elsif char == "("
+          LOOKAROUND.each { |prefix, name| return name if source[i + 1, prefix.length] == prefix }
+        elsif char == "+" && quantifier
+          return "a possessive quantifier"
+        end
+
+        quantifier = !in_class && %w[* + ?].include?(char) && !(char == "?" && source[i - 1] == "(")
+        i += 1
+      end
+
+      nil
+    end
+
+    # ActiveSupport::TimeWithZone and the like, without a dependency on them.
+    def time_like?(value)
+      value.respond_to?(:utc) && value.respond_to?(:iso8601) && !value.is_a?(Date)
+    end
+  end
+end

data/lib/ndb/connection.rb

@@ -0,0 +1,284 @@
+# frozen_string_literal: true
+
+require "socket"
+require "uri"
+
+module Ndb
+  Response = Struct.new(:status, :headers, :body)
+
+  # Where the daemon listens: a unix socket path, or a TCP host and port.
+  # The host name is resolved each time a connection opens, so a changed
+  # DNS record is followed after a reconnect.
+  class Endpoint
+    attr_reader :path, :host, :port
+
+    # "unix:///path", "tcp://host:port", or a bare socket path.
+    def self.parse(address)
+      address = address.to_s
+
+      return new(path: address) if address.start_with?("/")
+
+      uri = URI.parse(address)
+
+      case uri.scheme
+      when "unix" then new(path: uri.path)
+      when "tcp" then new(host: uri.host, port: uri.port)
+      else raise ArgumentError, "ndb address #{address.inspect} must be unix:///path or tcp://host:port"
+      end
+    rescue URI::InvalidURIError
+      raise ArgumentError, "ndb address #{address.inspect} must be unix:///path or tcp://host:port"
+    end
+
+    def initialize(path: nil, host: nil, port: nil)
+      if path.to_s.empty? && (host.to_s.empty? || port.nil?)
+        raise ArgumentError, "ndb address needs a socket path or a host and port"
+      end
+
+      @path = path
+      @host = host
+      @port = port
+    end
+
+    def unix? = !@path.to_s.empty?
+
+    def to_s = unix? ? "unix://#{@path}" : "tcp://#{@host}:#{@port}"
+
+    def host_header = unix? ? "ndb" : "#{@host}:#{@port}"
+
+    def connect(timeout)
+      return UNIXSocket.new(@path) if unix?
+
+      TCPSocket.new(@host, @port, connect_timeout: timeout).tap { |sock| sock.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1) }
+    end
+  end
+
+  # One HTTP/1.1 keep-alive connection on a unix socket or TCP. Not
+  # thread-safe: the Pool hands one connection to one thread at a time.
+  class Connection
+    STREAM_CHUNK = 256 * 1024
+
+    attr_reader :socket
+
+    def initialize(endpoint, timeout)
+      @endpoint = endpoint
+      @timeout = timeout
+      @socket = nil
+    end
+
+    # body is a String, or an IO that is sent chunked and never retried. A
+    # dropped reused connection is sent again only when idempotent is true
+    # (a read): a write may already have been processed.
+    def request(method, target, body, headers: {}, idempotent: false)
+      reused = !@socket.nil?
+
+      begin
+        send_request(method, target, body, headers)
+        read_response
+      rescue EOFError, Errno::EPIPE, Errno::ECONNRESET, IOError
+        close
+
+        raise unless reused && idempotent && !body.respond_to?(:read)
+
+        reused = false
+
+        retry
+      end
+    rescue SystemCallError, EOFError, IOError => e
+      close
+
+      raise ConnectionError, "cannot reach ndb at #{@endpoint}: #{e.message}"
+    rescue IO::TimeoutError
+      close
+
+      raise TimeoutError, "ndb did not answer within #{@timeout}s"
+    end
+
+    # Sends the request and yields the body of a 2xx response chunk by chunk,
+    # as it arrives (a watch stream). The connection is closed at the end: a
+    # stream is never reused. An error response is read whole and returned.
+    def stream(method, target, body, headers: {}, &)
+      close
+
+      send_request(method, target, body, headers)
+
+      status = @socket.gets("\r\n") or raise EOFError
+      response_headers = read_headers
+      code = status.split(" ", 3)[1].to_i
+
+      return Response.new(code, response_headers, read_body(response_headers)) if code >= 300
+
+      each_body_chunk(response_headers, &)
+
+      Response.new(code, response_headers, nil)
+    rescue SystemCallError, EOFError, IOError => e
+      raise ConnectionError, "cannot reach ndb at #{@endpoint}: #{e.message}"
+    rescue IO::TimeoutError
+      raise TimeoutError, "ndb did not answer within #{@timeout}s"
+    ensure
+      close
+    end
+
+    def close
+      @socket&.close
+    rescue IOError
+      nil
+    ensure
+      @socket = nil
+    end
+
+    private
+
+    def open_socket
+      @socket = @endpoint.connect(@timeout)
+      @socket.timeout = @timeout
+    rescue SystemCallError, SocketError => e
+      raise ConnectError, "cannot reach ndb at #{@endpoint}: #{e.message}"
+    end
+
+    def send_request(method, target, body, headers)
+      open_socket unless @socket
+
+      head = +"#{method} #{target} HTTP/1.1\r\nHost: #{@endpoint.host_header}\r\nAccept: application/json\r\n"
+
+      { "Content-Type" => "application/json" }.merge(headers).each { |name, value| head << "#{name}: #{value}\r\n" }
+
+      return send_streamed(head, body) if body.respond_to?(:read)
+
+      head << "Content-Length: #{body.bytesize}\r\n\r\n"
+
+      @socket.write(head, body)
+    end
+
+    def send_streamed(head, io)
+      @socket.write(head, "Transfer-Encoding: chunked\r\n\r\n")
+
+      while (chunk = io.read(STREAM_CHUNK))
+        @socket.write(chunk.bytesize.to_s(16), "\r\n", chunk, "\r\n")
+      end
+
+      @socket.write("0\r\n\r\n")
+    end
+
+    def read_response
+      status = @socket.gets("\r\n") or raise EOFError
+      headers = read_headers
+      body = read_body(headers)
+
+      close if headers["connection"].to_s.casecmp?("close")
+
+      Response.new(status.split(" ", 3)[1].to_i, headers, body)
+    end
+
+    def read_headers
+      headers = {}
+
+      while (line = @socket.gets("\r\n")) && line != "\r\n"
+        name, value = line.split(":", 2)
+        headers[name.downcase] = value.to_s.strip
+      end
+
+      headers
+    end
+
+    def read_body(headers)
+      return read_chunked if headers["transfer-encoding"].to_s.casecmp?("chunked")
+      return (@socket.read(headers["content-length"].to_i) || +"") if headers["content-length"]
+
+      body = @socket.read.to_s
+      close
+
+      body
+    end
+
+    def each_body_chunk(headers)
+      return yield(read_body(headers)) unless headers["transfer-encoding"].to_s.casecmp?("chunked")
+
+      loop do
+        size = @socket.gets("\r\n").to_s.to_i(16)
+
+        break if size.zero?
+
+        yield @socket.read(size)
+        @socket.read(2)
+      end
+    end
+
+    def read_chunked
+      body = +""
+
+      loop do
+        size = @socket.gets("\r\n").to_s.to_i(16)
+
+        break if size.zero?
+
+        body << @socket.read(size)
+        @socket.read(2)
+      end
+
+      @socket.gets("\r\n")
+
+      body
+    end
+  end
+
+  # A small thread-safe pool: one connection per in-flight request, idle
+  # ones are kept for keep-alive. It drops its connections after a fork.
+  class Pool
+    attr_reader :created
+
+    def initialize(endpoint, timeout:, size:)
+      @endpoint = endpoint
+      @timeout = timeout
+      @size = size
+      @mutex = Mutex.new
+      @idle = []
+      @created = 0
+      @pid = Process.pid
+    end
+
+    def with
+      conn = checkout
+
+      begin
+        yield conn
+      rescue StandardError
+        conn.close
+
+        raise
+      ensure
+        checkin(conn)
+      end
+    end
+
+    def idle_connections
+      @mutex.synchronize { @idle.dup }
+    end
+
+    private
+
+    def checkout
+      @mutex.synchronize do
+        reset_after_fork
+
+        return @idle.pop if @idle.any?
+
+        @created += 1
+
+        Connection.new(@endpoint, @timeout)
+      end
+    end
+
+    def checkin(conn)
+      @mutex.synchronize do
+        @idle << conn if @idle.size < @size
+      end
+    end
+
+    def reset_after_fork
+      return if @pid == Process.pid
+
+      @idle.clear
+      @pid = Process.pid
+    end
+  end
+end

data/lib/ndb/errors.rb

@@ -0,0 +1,67 @@
+# frozen_string_literal: true
+
+module Ndb
+  # An error the daemon answered (code + HTTP status), or a transport failure.
+  class Error < StandardError
+    attr_reader :code, :status
+
+    def initialize(message = nil, code: nil, status: nil)
+      super(message)
+
+      @code = code
+      @status = status
+    end
+  end
+
+  class ConnectionError < Error; end
+  class TimeoutError < ConnectionError; end
+
+  # The connect failed: the socket is missing or refuses. Nothing was sent.
+  class ConnectError < ConnectionError; end
+
+  # The daemon is up but still recovering (HTTP 503). Nothing was done.
+  class NotReady < Error; end
+
+  class Unauthorized < Error; end
+  class NotFound < Error; end
+  class UnknownCollection < Error; end
+  class VersionConflict < Error; end
+  class UniqueViolation < Error; end
+  class CatalogConflict < Error; end
+  class CatalogInvalid < Error; end
+  class ReadOnlyCollection < Error; end
+  class InvalidQuery < Error; end
+  class InvalidCursor < Error; end
+
+  # The consumer was too slow: call watch again, it resumes from the last ack.
+  class WatchLagged < Error; end
+
+  # The WAL files that hold the history were deleted (wal_retention): the
+  # consumer cannot resume from its position and must start over.
+  class HistoryTruncated < Error; end
+
+  # The daemon error code -> the Ruby class.
+  CODE_TO_ERROR = {
+    "unauthorized" => Unauthorized,
+    "not_ready" => NotReady,
+    "not_found" => NotFound,
+    "unknown_collection" => UnknownCollection,
+    "version_conflict" => VersionConflict,
+    "unique_violation" => UniqueViolation,
+    "duplicate_id" => UniqueViolation,
+    "catalog_conflict" => CatalogConflict,
+    "catalog_invalid" => CatalogInvalid,
+    "direct_collection" => ReadOnlyCollection,
+    "invalid_query" => InvalidQuery,
+    "invalid_request" => InvalidQuery,
+    "cursor_invalid" => InvalidCursor,
+    "cursor_mismatch" => InvalidCursor,
+    "cursor_expired" => InvalidCursor,
+    "watch_lagged" => WatchLagged,
+    "history_truncated" => HistoryTruncated
+  }.freeze
+
+  def self.error_for(code, message, status)
+    CODE_TO_ERROR.fetch(code, Error).new("#{code}: #{message}", code: code, status: status)
+  end
+end

data/lib/ndb/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module Ndb
+  VERSION = "0.1.0"
+end

data/lib/ndb.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+require "ndb/version"
+require "ndb/errors"
+require "ndb/conditions"
+require "ndb/connection"
+require "ndb/client"

metadata

@@ -0,0 +1,53 @@
+--- !ruby/object:Gem::Specification
+name: ndb
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- thadeu
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies: []
+description: 'Ndb::Client talks HTTP to ndb over a unix socket or TCP: collections,
+  documents with versions, queries, aggregates, watch streams and blobs. Stdlib only,
+  no runtime dependencies. The daemon decides every query semantic.'
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE
+- README.md
+- lib/ndb.rb
+- lib/ndb/client.rb
+- lib/ndb/conditions.rb
+- lib/ndb/connection.rb
+- lib/ndb/errors.rb
+- lib/ndb/version.rb
+homepage: https://github.com/thadeu/ndb
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/thadeu/ndb
+  source_code_uri: https://github.com/thadeu/ndb/tree/main/sdk/ruby
+  changelog_uri: https://github.com/thadeu/ndb/blob/main/CHANGELOG.md
+  bug_tracker_uri: https://github.com/thadeu/ndb/issues
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.2'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.9
+specification_version: 4
+summary: Plain Ruby client for the ndb NDJSON query daemon
+test_files: []