ndb-rails 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 407a37e97a7c0f50f39d4ca365e59bd9ecec788a9fe357e4ff4fbc61add79962
+  data.tar.gz: 0de5474707d6bf68b55d1a44f56b612a8d5a59bc24cd7e96ff8066af00e98f8b
+SHA512:
+  metadata.gz: 5eeed45c9f972b596dd588122600f1f7729ca07957ca771a8af0d0ca34085aa8b3c8e76d898a9605e034a6402c8370bdf57e209b2da5320a84d5904176fbeb50
+  data.tar.gz: d52fa0af1af45659950dd152a68a5561aa311b98c7b3296f70df1185d6071b70740eae7f95763bb58d1aafc35984a6aa11cf04b822d05ab98ed2a02de208a88b

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 thadeu
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,343 @@
+# ndb-rails
+
+<img src="https://raw.githubusercontent.com/thadeu/ndb/main/brand/png/lockup-on-light.png" width="220">
+
+ActiveModel layer for [ndb](https://github.com/thadeu/ndb), an NDJSON query daemon. It gives you `Ndb::Record` (a model class like ActiveRecord), a lazy relation, validations, blobs and a Rails railtie. It sits on the plain client gem [`ndb`](https://github.com/thadeu/ndb/tree/main/sdk/ruby) (`Ndb::Client`), which does the HTTP over a unix socket or TCP. The gem only builds JSON. The daemon decides what every query means.
+
+No Rails, or no models? Use the gem `ndb` alone: it has no runtime dependency.
+
+Docs: [client guide](https://github.com/thadeu/ndb/blob/main/docs/dev/clients.md), [query language](https://github.com/thadeu/ndb/blob/main/docs/dev/query.md), [HTTP API](https://github.com/thadeu/ndb/blob/main/docs/dev/http-api.md).
+
+## Install
+
+Needs Ruby 3.2 or newer, `activemodel` and `activesupport` 7.1 or newer, and a running ndb daemon. The gem `ndb` (`~> 0.1`) is installed with it, as a dependency.
+
+```ruby
+# Gemfile
+gem "ndb-rails"
+```
+
+```sh
+bundle add ndb-rails
+# or: gem install ndb-rails
+```
+
+In Rails, the gem loads a railtie. It reads `config/ndb.yml` and, when the app eager loads (production), it declares the collection of every model at boot. In development a model declares its collection on first use. Without Bundler autorequire, use `require "ndb/rails"`. Without Rails, call `Ndb::Rails.configure` yourself (below).
+
+## Connect
+
+Rails: create `config/ndb.yml`. It has one block per environment. ERB and YAML anchors work.
+
+```yaml
+default: &default
+  timeout: 30          # seconds, per request
+  pool_size: 16        # idle connections kept
+  ready_timeout: 10    # seconds to retry while the daemon starts; 0 turns it off
+
+development:
+  <<: *default
+  socket: /tmp/ndb.sock
+
+production:
+  <<: *default
+  url: tcp://ndb.internal:7070
+  token: <%= ENV["NDB_TOKEN"] %>
+```
+
+Keys: `url`, `socket`, `token`, `timeout`, `pool_size`, `ready_timeout`. Any other key is ignored.
+
+Any app: configure in code, for example in `config/initializers/ndb.rb`.
+
+```ruby
+Ndb::Rails.configure do |c|
+  c.socket = "/run/ndb/ndb.sock"            # or: c.url = "tcp://10.0.0.5:7070"
+  c.token = ENV["NDB_TOKEN"]
+  c.timeout = 30
+  c.pool_size = 16
+  c.ready_timeout = 10
+end
+```
+
+- The address is, in order: `url`, `socket`, env `NDB_URL`, then the default socket `/var/run/ndb/ndb.sock`.
+- The address is `unix:///path`, `tcp://host:port`, or a bare path that starts with `/`.
+- The token is `token`, or env `NDB_TOKEN`. It goes only in the `Authorization` header. A unix socket does not need it.
+- `ready_timeout` is how long a call retries while the daemon is not ready (`not_ready`, or the socket is missing or refuses). It uses backoff with jitter. A write is retried only when the daemon certainly did not process it. A read is also retried after a dropped connection. A timeout is never retried.
+- The pool drops its connections after a `fork`.
+- `Ndb::Rails.client` is an `Ndb::Client` built from this configuration (`Ndb::Rails.configuration`). You can also build one: `Ndb::Client.new(socket: "/tmp/ndb.sock", token: nil, timeout: 30, pool_size: 16, ready_timeout: 10)`. The [`ndb` README](https://github.com/thadeu/ndb/blob/main/sdk/ruby/README.md) documents every client call.
+
+## Define a model
+
+```ruby
+class Call < Ndb::Record
+  primary_key :call_id
+
+  attribute :status, :string, index: true
+  attribute :caller, :string, unique: true
+  attribute :duration, :integer
+  attribute :at, :datetime, time: true
+  attribute :expires_at, :datetime, expires: true
+  attribute :recording, :blob
+
+  validates :status, presence: true
+end
+```
+
+| Option or method | Meaning |
+|---|---|
+| `primary_key :call_id` | The id field. Default is `_id` (a string, made by the daemon). |
+| `index: true` | Index this field. |
+| `unique: true` | Unique value. A second one raises `RecordNotUnique`. |
+| `time: true` | The collection time field. Used by zone maps, `$dateTrunc` and TTL. |
+| `expires: true` | The field holds the time the document expires. |
+| `ttl 24.hours` | Documents expire this long after the time field. Use `ttl` or `expires:`, not both: the daemon returns `catalog_invalid`. |
+| `self.collection_name = "x"` | Collection name. Default follows Rails naming (`Call` becomes `calls`). |
+| `source "glob"` | Declares a read-only direct-mode collection over files. Writes raise `ReadOnlyCollection`. |
+
+- Attributes are ActiveModel attributes, so every ActiveModel type works (`:string`, `:integer`, `:float`, `:boolean`, `:datetime`, `:date`, and so on). Times are sent as RFC 3339 UTC strings with milliseconds. `:blob` is added by this gem.
+- Validations, callbacks (`save`, `create`, `update`, `destroy`), dirty tracking and `as_json` work as in ActiveModel.
+- The model declares its collection on first use, once per process.
+- Mark a base class with `self.abstract_class = true` to share settings.
+
+## Insert, find, update, delete
+
+```ruby
+call = Call.create!(call_id: "c1", status: "answered", caller: "a", at: Time.now.utc)
+Call.create!([{ call_id: "c2", status: "failed", caller: "b" }, { call_id: "c3", status: "answered", caller: "c" }])
+
+call = Call.new(call_id: "c4", status: "answered", caller: "d")
+call.save          # false when invalid
+call.save!         # raises Ndb::RecordInvalid
+
+call = Call.find("c1")        # raises Ndb::NotFound
+Call.find_by(status: "failed") # nil when missing
+Call.find_by!(status: "x")     # raises NotFound
+
+call.update!(status: "done")  # sends only the changed fields
+call.version                   # the document version
+
+call.destroy
+```
+
+- Every load asks for the version (`_v`). `save`, `update` and `destroy` send it back as `if_version`. If another writer changed the document first, you get `Ndb::StaleObjectError` (code `version_conflict`).
+- Setting an attribute to `nil` and saving removes the field from the document.
+- A duplicate id raises `RecordNotUnique` (code `duplicate_id`).
+- `persisted?`, `new_record?`, `destroyed?`, `id` and `version` work as in ActiveRecord.
+
+```ruby
+begin
+  call.update!(status: "done")
+rescue Ndb::StaleObjectError
+  call = Call.find(call.id)
+  retry
+end
+```
+
+## Query
+
+Relations are lazy and immutable. The request goes out when you call `to_a`, `first`, `each`, `count` and so on.
+
+```ruby
+Call.where(status: "answered")
+Call.where(status: %w[answered failed])      # $in
+Call.where(duration: 30..60)                 # $gte and $lte
+Call.where(duration: 30..)                   # $gte
+Call.where(duration: nil)                    # $exists: false
+Call.where(at: 1.day.ago..)
+Call.where("agent.id" => 7)                  # dot path
+Call.where(duration: { "$gt" => 10 })        # any daemon operator, as a Hash
+Call.where.not(status: "failed")             # $ne, $nin, $exists
+Call.where(status: "failed").or(Call.where(duration: 60..))
+Call.where("$or" => [{ status: "failed" }, { duration: { "$gt" => 50 } }])
+
+Call.where(status: "answered")
+    .order(at: :desc, call_id: :asc)
+    .limit(50)
+    .offset(100)
+    .to_a
+
+Call.where(status: "answered").pluck(:call_id)         # one field: an array of values
+Call.where(status: "answered").pluck(:call_id, :status) # several: an array of arrays
+Call.where(status: "answered").first(5)
+Call.where(status: "failed").exists?
+```
+
+- Chained `where` calls are ANDed. `$regex` does not exist in the daemon yet. See the [query language](https://github.com/thadeu/ndb/blob/main/docs/dev/query.md) for the full operator list.
+- The order of the `order` keys is the order of the sort. The id is always the last key.
+- `page(n).per(size)` sets offset paging. The default page size is 25.
+- `pluck` asks the daemon for only those fields.
+- There is no `explain` on a relation. Use the client: `Ndb::Rails.client.find(coll, "where" => {...}, "explain" => true)` returns a `Page` with `plan` (the `X-Ndb-Plan` header: `id`, `index:<field>`, `scan`, `snapshot`) and `examined`.
+
+### Regular expressions
+
+A Ruby `Regexp` works in `where`, `where.not` and `or`. The gem `ndb` turns it into `$regex` and `$options` (the daemon uses RE2). Ruby `^` and `$` match at each line, so `m` is always sent. `/i` is `i`, `/m` is `s`, `/x` is `x`. Look-around, backreferences, atomic groups, possessive quantifiers, `\h`, `\G`, `\R` and `\X` raise `Ndb::InvalidQuery` before the request.
+
+```ruby
+Call.where(caller: /^55/)
+Call.where(caller: /^55/i).where.not(status: /fail/)
+Call.where(caller: /^55/).or(Call.where(caller: /^11/))
+Call.where(caller: { "$regex" => "^55" })   # a String: no `m`, and a literal ^prefix uses the index
+```
+
+A Regexp is a scan filter. For a prefix search that uses the index, pass the String form. See the [`ndb` README](https://github.com/thadeu/ndb/blob/main/sdk/ruby/README.md) for the full mapping.
+
+### Count
+
+```ruby
+Call.where(status: "answered").count
+```
+
+### Cursor pagination
+
+```ruby
+relation = Call.where(status: "answered").order(:call_id).limit(100)
+rows = relation.to_a
+token = relation.next_cursor   # nil on the last page
+
+Call.where(status: "answered").order(:call_id).limit(100).after(token).to_a
+```
+
+A token works only with the same `where` and `order`. Another query shape raises `InvalidCursor`.
+
+To walk every row, use `find_each`. It follows the cursor in batches (default 1000):
+
+```ruby
+Call.where(status: "answered").find_each(batch_size: 500) { |call| puts call.call_id }
+
+Call.in_batches(of: 500) { |batch| batch.each { |call| ... } }
+```
+
+## Aggregate
+
+Relations have simple group calculations. Each one is one aggregate request.
+
+```ruby
+Call.group(:status).count            # => {"answered" => 2, "failed" => 1}
+Call.group(:status).sum(:duration)
+Call.group(:status, :caller).count   # keys are arrays
+Call.where(status: "answered").average(:duration)
+Call.minimum(:duration)
+Call.maximum(:duration)
+Call.sum(:duration)
+```
+
+For a full pipeline, use the client. The pipeline needs exactly one `$group`. Accumulators: `$count`, `$sum`, `$avg`, `$min`, `$max`, `$first`, `$last`.
+
+```ruby
+page = Ndb::Rails.client.aggregate(Call.collection_name,
+  pipeline: [
+    { "$match" => { "duration" => { "$gte" => 1 } } },
+    { "$group" => {
+      "_id" => {
+        "status" => "$status",
+        "hour" => { "$dateTrunc" => { "date" => "$at", "unit" => "hour" } }
+      },
+      "n" => { "$count" => {} },
+      "avg" => { "$avg" => "$duration" }
+    } },
+    { "$sort" => { "n" => -1 } }
+  ],
+  limit: 100,
+  count: true)
+
+page.rows         # array of Hash
+page.total        # with count: true
+page.next_cursor  # nil on the last page
+```
+
+`$dateTrunc` units are `minute`, `hour`, `day`, `week` and `month`. Optional `"timezone"`: a name such as `"Europe/Rome"`.
+
+## Watch
+
+A watch streams the changes of a collection to a named consumer: first the events after its last ack, then new events as they happen. The watch API is on the client (`Ndb::Client`).
+
+```ruby
+client = Ndb::Rails.client
+
+begin
+  client.watch(Call.collection_name, "relay", where: { status: "answered" }) do |event|
+    # event has "pos", "op" ("put" or "del"), "id", "v" and "doc"
+    handle(event)
+
+    client.ack(Call.collection_name, "relay", event["pos"])
+  end
+rescue Ndb::WatchLagged
+  retry # the consumer was too slow. watch continues after the last ack.
+rescue Ndb::HistoryTruncated
+  raise # the saved position is too old. Start over.
+end
+```
+
+- `where:` takes the same conditions as `where` on a model (Hash and Range). Only matching events come. A delete event has no `"doc"`; it is sent only if this stream saw an earlier matching put of that id.
+- `from: "now"` starts a new consumer at the end. `limit: n` ends the stream after n events.
+- Without a block, `watch` returns an Enumerator.
+- Delivery is at least once. After a restart an event can come again, so make the work safe to repeat.
+- `HistoryTruncated` means the WAL files for the position were deleted (`wal_retention`).
+- A watch holds one connection until it ends.
+
+## Blobs
+
+Large values live in the blob store. The document keeps only a reference (`$blob`, `size`, `type`).
+
+```ruby
+call = Call.new(call_id: "c1", status: "answered", caller: "a",
+                recording: File.open("call.wav"))   # an IO or a String
+call.save!                                          # uploads the blob, then saves
+
+call.recording.ref           # "sha256:..."
+call.recording.size
+
+loaded = Call.find("c1")
+loaded.recording.read        # all the bytes
+loaded.recording.each_chunk { |chunk| out.write(chunk) } # 1 MiB ranges
+```
+
+A blob behaves like a read-only IO (`read`, `each_chunk`, `eof?`, `rewind`). A loaded blob reads the daemon in 1 MiB ranges, so memory stays small. The same content always gives the same reference. You can also use `Ndb::Blob.new(io_or_string, content_type: "audio/wav")`.
+
+## Errors
+
+The transport and daemon errors are defined by the gem `ndb`. They all live in the `Ndb` module (`Ndb::Error`, `Ndb::NotFound`, ...). For an ActiveRecord-like `rescue`, this gem adds two aliases: `Ndb::StaleObjectError` is `Ndb::VersionConflict`, and `Ndb::RecordNotUnique` is `Ndb::UniqueViolation`. They are the same classes, so a `rescue` of either name catches the same error. `Ndb::RecordInvalid` and `Ndb::RecordNotSaved` exist only here, and they inherit from `Ndb::Error`. Every error has `code` (the daemon code) and `status` (the HTTP status).
+
+| Class | Code |
+|---|---|
+| `Unauthorized` | `unauthorized` |
+| `NotReady` | `not_ready` |
+| `NotFound` | `not_found` |
+| `UnknownCollection` | `unknown_collection` |
+| `StaleObjectError` | `version_conflict` |
+| `RecordNotUnique` | `unique_violation`, `duplicate_id` |
+| `CatalogConflict` | `catalog_conflict` |
+| `CatalogInvalid` | `catalog_invalid` |
+| `ReadOnlyCollection` | `direct_collection` |
+| `InvalidQuery` | `invalid_query`, `invalid_request` |
+| `InvalidCursor` | `cursor_invalid`, `cursor_mismatch`, `cursor_expired` |
+| `WatchLagged` | `watch_lagged` |
+| `HistoryTruncated` | `history_truncated` |
+| `RecordInvalid` | ActiveModel validation failed (has `record`) |
+| `RecordNotSaved` | `save!` returned false without errors |
+| `ConnectionError` | cannot reach the daemon. Subclasses: `ConnectError` (nothing was sent), `TimeoutError` |
+
+Any other code is a plain `Ndb::Error`. A response that is not a daemon error has the code `http_status`.
+
+```ruby
+rescue Ndb::RecordNotUnique => e
+  e.code    # "unique_violation"
+  e.status  # 409
+```
+
+## Testing
+
+The tests of this gem build the daemon from the repository (`go build ./cmd/ndb`), start it on a short temp unix socket (macOS allows 104 bytes), wait for `GET /health` to return 200, and stop it at exit. To do the same by hand:
+
+```sh
+ndb -config /nonexistent.yml -data-dir "$(mktemp -d)" -unix /tmp/ndb-test.sock
+```
+
+```ruby
+Ndb::Rails.configure { |c| c.socket = "/tmp/ndb-test.sock" }
+```
+
+Use a new collection name per test (`self.collection_name = ...`). Run the suite with `bundle exec rake test` in `sdk/rails`. Its Gemfile uses the gem `ndb` from `../ruby`.
+
+## License
+
+MIT.

data/lib/ndb/rails/blob.rb

@@ -0,0 +1,145 @@
+# frozen_string_literal: true
+
+require "stringio"
+
+module Ndb
+  # A large value in the blob store (plan §7.5). The document keeps only
+  # {"$blob" => "sha256:...", "size" => N, "type" => "..."}.
+  #
+  # It behaves like a read-only IO. A blob built from an IO or a String is
+  # pending: it is uploaded when the record is saved. A blob from a loaded
+  # record reads the daemon in 1 MiB ranges, so memory stays small.
+  class Blob
+    CHUNK_SIZE = 1 << 20
+
+    attr_reader :ref, :size, :content_type
+
+    def self.from_reference(reference)
+      blob = allocate
+
+      blob.send(:stored!, reference["$blob"], reference["size"], reference["type"])
+
+      blob
+    end
+
+    def initialize(source, content_type: nil)
+      @source = source.is_a?(String) ? StringIO.new(source) : source
+      @content_type = content_type || (source.respond_to?(:content_type) ? source.content_type : nil)
+      @pos = 0
+    end
+
+    def uploaded?
+      !@ref.nil?
+    end
+
+    # Uploads a pending blob; a stored blob does nothing.
+    def upload!(client = Rails.client)
+      return self if uploaded?
+
+      @source.rewind if @source.respond_to?(:rewind)
+
+      stored = client.put_blob(@source, @content_type)
+
+      stored!(stored["$blob"], stored["size"], stored["type"])
+    end
+
+    # The document value, uploading first if needed.
+    def to_reference
+      upload!
+
+      { "$blob" => ref, "size" => size, "type" => content_type }.compact
+    end
+
+    def as_json(*)
+      uploaded? ? to_reference : nil
+    end
+
+    # IO#read: all the rest, or up to length bytes (nil at the end).
+    def read(length = nil)
+      return @source.read(length) unless uploaded?
+      return read_all if length.nil?
+      return +"" if length.zero?
+      return nil if eof?
+
+      fetch([length, size - @pos].min)
+    end
+
+    def each_chunk(chunk_size = CHUNK_SIZE)
+      return enum_for(:each_chunk, chunk_size) unless block_given?
+
+      while (chunk = read(chunk_size))
+        yield chunk
+      end
+    end
+
+    def eof?
+      uploaded? ? @pos >= size : @source.eof?
+    end
+
+    def rewind
+      @pos = 0
+
+      @source.rewind if !uploaded? && @source.respond_to?(:rewind)
+
+      0
+    end
+
+    def ==(other)
+      other.is_a?(Blob) && uploaded? && other.uploaded? && ref == other.ref
+    end
+
+    private
+
+    def stored!(ref, size, content_type)
+      @ref = ref
+      @size = size
+      @content_type = content_type
+      @source = nil
+      @pos = 0
+
+      self
+    end
+
+    def read_all
+      out = +""
+
+      each_chunk { |chunk| out << chunk }
+
+      out
+    end
+
+    def fetch(count)
+      data = Rails.client.get_blob(ref, from: @pos, to: @pos + count - 1)
+
+      @pos += data.bytesize
+
+      data
+    end
+  end
+
+  # attribute :recording, :blob
+  class BlobType < ActiveModel::Type::Value
+    def type
+      :blob
+    end
+
+    def cast(value)
+      case value
+      when nil, Blob then value
+      when Hash then value.key?("$blob") ? Blob.from_reference(value) : nil
+      when String then Blob.new(value)
+      else value.respond_to?(:read) ? Blob.new(value) : raise(ArgumentError, "cannot make a blob from #{value.class}")
+      end
+    end
+
+    def deserialize(value)
+      cast(value)
+    end
+
+    def serialize(value)
+      value&.to_reference
+    end
+  end
+
+  ActiveModel::Type.register(:blob, BlobType)
+end

data/lib/ndb/rails/configuration.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+require "erb"
+require "yaml"
+
+module Ndb
+  module Rails
+    # The connection only: the address (url or socket), the token and the
+    # timeouts. ready_timeout is how many seconds a call retries while the
+    # daemon starts or recovers (0 turns it off). Unset address and token fall back to NDB_URL and NDB_TOKEN.
+    class Configuration
+      attr_accessor :url, :socket, :token, :timeout, :pool_size, :ready_timeout
+
+      def initialize
+        @timeout = 30
+        @pool_size = 16
+        @ready_timeout = 10
+      end
+
+      # Reads config/ndb.yml: one block per environment.
+      def load_file(path, env)
+        raw = YAML.safe_load(ERB.new(File.read(path)).result, aliases: true) || {}
+        settings = raw.fetch(env.to_s, {})
+
+        @url = settings["url"] if settings.key?("url")
+        @socket = settings["socket"] if settings.key?("socket")
+        @token = settings["token"] if settings.key?("token")
+        @timeout = settings["timeout"] if settings.key?("timeout")
+        @pool_size = settings["pool_size"] if settings.key?("pool_size")
+        @ready_timeout = settings["ready_timeout"] if settings.key?("ready_timeout")
+
+        self
+      end
+    end
+  end
+end

data/lib/ndb/rails/errors.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module Ndb
+  # ActiveRecord-like names for two daemon errors. They are the same classes,
+  # so a rescue of either name catches the same error.
+  StaleObjectError = VersionConflict
+  RecordNotUnique = UniqueViolation
+
+  # Raised by save!/create!/update! when validations fail.
+  class RecordInvalid < Error
+    attr_reader :record
+
+    def initialize(record)
+      @record = record
+
+      super("Validation failed: #{record.errors.full_messages.join(', ')}")
+    end
+  end
+end

data/lib/ndb/rails/persistence.rb

@@ -0,0 +1,170 @@
+# frozen_string_literal: true
+
+module Ndb
+  class RecordNotSaved < Error; end
+
+  # Writes: dirty attributes -> set/unset, the loaded version -> if_version.
+  module Persistence
+    extend ActiveSupport::Concern
+
+    included do
+      define_model_callbacks :save, :create, :update, :destroy
+    end
+
+    class_methods do
+      def create(attrs = {}, &block)
+        return attrs.map { |a| create(a, &block) } if attrs.is_a?(Array)
+
+        record = new(attrs)
+
+        yield record if block
+        record.save
+
+        record
+      end
+
+      def create!(attrs = {}, &block)
+        return attrs.map { |a| create!(a, &block) } if attrs.is_a?(Array)
+
+        record = new(attrs)
+
+        yield record if block
+        record.save!
+
+        record
+      end
+
+      def instantiate(row)
+        row = row.dup
+        version = row.delete("_v")
+        record = new(row.slice(*attribute_names))
+
+        record.send(:loaded!, version)
+
+        record
+      end
+    end
+
+    attr_reader :version
+
+    def id
+      public_send(self.class.primary_key)
+    end
+
+    def id=(value)
+      public_send("#{self.class.primary_key}=", value)
+    end
+
+    def persisted?
+      @persisted == true && !destroyed?
+    end
+
+    def new_record?
+      !@persisted
+    end
+
+    def destroyed?
+      @destroyed == true
+    end
+
+    def to_key
+      persisted? ? [id] : nil
+    end
+
+    def save(validate: true)
+      return false if validate && !valid?(new_record? ? :create : :update)
+
+      run_callbacks(:save) { new_record? ? create_record : update_record } ? true : false
+    end
+
+    def save!(**)
+      save(**) || raise(errors.any? ? RecordInvalid.new(self) : RecordNotSaved.new("Failed to save the record"))
+    end
+
+    def update(attrs)
+      assign_attributes(attrs)
+
+      save
+    end
+
+    def update!(attrs)
+      assign_attributes(attrs)
+
+      save!
+    end
+
+    def destroy
+      result = run_callbacks(:destroy) do
+        self.class.ndb_delete(id, if_version: version) if persisted?
+        @destroyed = true
+      end
+
+      result ? self : false
+    end
+
+    def ==(other)
+      other.instance_of?(self.class) && !id.nil? && id == other.id
+    end
+
+    private
+
+    def loaded!(version)
+      @persisted = true
+      @version = version
+
+      changes_applied
+    end
+
+    def create_record
+      run_callbacks(:create) do
+        result = self.class.ndb_insert(insert_document)
+
+        self.id = result["id"] if id.nil?
+        loaded!(result["v"])
+
+        true
+      end
+    end
+
+    def update_record
+      run_callbacks(:update) do
+        set, unset = changed_fields
+
+        unless set.empty? && unset.empty?
+          result = self.class.ndb_update(id, set: set, unset: unset, if_version: version)
+
+          @version = result["v"]
+        end
+
+        changes_applied
+
+        true
+      end
+    end
+
+    def insert_document
+      attribute_names.each_with_object({}) do |name, doc|
+        value = json_attribute(name)
+
+        doc[name] = value unless value.nil?
+      end
+    end
+
+    def changed_fields
+      names = changed - [self.class.primary_key]
+      set = {}
+      unset = []
+
+      names.each do |name|
+        value = json_attribute(name)
+        value.nil? ? unset << name : set[name] = value
+      end
+
+      [set, unset]
+    end
+
+    def json_attribute(name)
+      Conditions.json_value(self.class.attribute_types[name].serialize(public_send(name)))
+    end
+  end
+end

data/lib/ndb/rails/railtie.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+require "rails/railtie"
+
+module Ndb
+  module Rails
+    # Reads config/ndb.yml (the connection only) and declares the collection
+    # of every model on boot when the app eager loads (production). In
+    # development a model declares its collection on first use.
+    class Railtie < ::Rails::Railtie
+      initializer "ndb.configure" do |app|
+        path = app.root.join("config", "ndb.yml")
+
+        if path.exist?
+          Ndb::Rails.configuration.load_file(path, ::Rails.env)
+          Ndb::Rails.configure { |_| }
+        end
+      end
+
+      config.after_initialize do |app|
+        if app.config.eager_load
+          Ndb::Record.descendants.reject(&:abstract_class?).each(&:ensure_collection!)
+        end
+      end
+    end
+  end
+end

data/lib/ndb/rails/record.rb

@@ -0,0 +1,182 @@
+# frozen_string_literal: true
+
+module Ndb
+  # The model class, like ActiveRecord::Base: ActiveModel + a lazy Relation.
+  class Record
+    include ActiveModel::Model
+    include ActiveModel::Attributes
+    include ActiveModel::Validations
+    include ActiveModel::Callbacks
+    include ActiveModel::Dirty
+    include ActiveModel::Serializers::JSON
+    include Persistence
+
+    extend ActiveSupport::DescendantsTracker
+
+    ENSURE_LOCK = Mutex.new
+
+    attribute :_id, :string
+
+    class << self
+      QUERY_METHODS = %i[
+        where order limit offset page after group find find_by find_by! first count pluck exists?
+        find_each in_batches sum average minimum maximum to_a each
+      ].freeze
+
+      delegate(*QUERY_METHODS, to: :all)
+
+      attr_writer :abstract_class
+
+      def abstract_class?
+        @abstract_class == true
+      end
+
+      def all
+        Relation.new(self)
+      end
+
+      def collection_name
+        @collection_name || model_name.plural
+      end
+
+      def collection_name=(name)
+        @collection_name = name.to_s
+      end
+
+      def primary_key(name = nil)
+        if name
+          @primary_key = name.to_s
+          attribute(name, :string) unless attribute_types.key?(@primary_key)
+        end
+
+        @primary_key || (superclass.respond_to?(:primary_key) ? superclass.primary_key : "_id")
+      end
+
+      def primary_key=(name)
+        primary_key(name)
+      end
+
+      def source(glob = nil)
+        @source = glob if glob
+
+        @source
+      end
+
+      def attribute(name, *args, index: false, unique: false, time: false, expires: false, **options)
+        own_indexed << name.to_s if index
+        own_unique << name.to_s if unique
+        @time_field = name.to_s if time
+        @expires_field = name.to_s if expires
+
+        super(name, *args, **options)
+      end
+
+      # The collection time field (plan §5): zone maps, $dateTrunc, TTL.
+      def time_field
+        @time_field || (superclass.respond_to?(:time_field) ? superclass.time_field : nil)
+      end
+
+      # The field that holds the expiry time of each document (plan §11.5).
+      def expires_field
+        @expires_field || (superclass.respond_to?(:expires_field) ? superclass.expires_field : nil)
+      end
+
+      # Documents expire this long after the time field: `ttl 24.hours`.
+      # Returns the Go-style duration the daemon reads, such as "86400s".
+      def ttl(duration = nil)
+        @ttl = "#{duration.to_i}s" if duration
+
+        @ttl || (superclass.respond_to?(:ttl) ? superclass.ttl : nil)
+      end
+
+      def indexed_fields
+        inherited_list(:indexed_fields) | own_indexed
+      end
+
+      def unique_fields
+        inherited_list(:unique_fields) | own_unique
+      end
+
+      # The ensure_collection body for this model.
+      def collection_definition
+        definition = { "id" => primary_key }
+
+        definition["indexes"] = indexed_fields if indexed_fields.any?
+        definition["unique"] = unique_fields if unique_fields.any?
+        definition["time"] = time_field if time_field
+        definition["expires_field"] = expires_field if expires_field
+        definition["ttl"] = ttl if ttl
+        definition.merge!("write" => "direct", "path" => source) if source
+
+        definition
+      end
+
+      # Declares the collection once per process (and once per client).
+      def ensure_collection!
+        client = Rails.client
+
+        return if @ensured_client.equal?(client)
+
+        ENSURE_LOCK.synchronize do
+          client.ensure_collection(collection_name, collection_definition)
+
+          @ensured_client = client
+        end
+      end
+
+      # Every load asks for "_v", so a loaded record can send if_version.
+      def ndb_find(body)
+        ensure_collection!
+
+        Rails.client.find(collection_name, body.merge("versions" => true))
+      end
+
+      def ndb_aggregate(body)
+        ensure_collection!
+
+        Rails.client.aggregate(collection_name, body)
+      end
+
+      def ndb_insert(doc)
+        ensure_collection!
+
+        Rails.client.insert(collection_name, [doc]).first
+      end
+
+      def ndb_update(id, **change)
+        ensure_collection!
+
+        Rails.client.update(collection_name, id, **change)
+      end
+
+      def ndb_delete(id, if_version: nil)
+        ensure_collection!
+
+        Rails.client.delete(collection_name, id, if_version: if_version)
+      end
+
+      private
+
+      def own_indexed
+        @own_indexed ||= []
+      end
+
+      def own_unique
+        @own_unique ||= []
+      end
+
+      def inherited_list(name)
+        superclass.respond_to?(name) ? superclass.public_send(name) : []
+      end
+    end
+
+    # The generated _id stays out of the JSON when another field is the id.
+    def serializable_hash(options = nil)
+      hash = super
+
+      hash.delete("_id") if self.class.primary_key != "_id" && hash["_id"].nil?
+
+      hash
+    end
+  end
+end

data/lib/ndb/rails/relation.rb

@@ -0,0 +1,286 @@
+# frozen_string_literal: true
+
+module Ndb
+  # A lazy, immutable query. Every builder returns a new relation; the
+  # request goes to the daemon on each, to_a, first, count, pluck, ...
+  class Relation
+    include Enumerable
+
+    DEFAULT_PER_PAGE = 25
+    BATCH_SIZE = 1000
+    EMPTY = { where: [].freeze, order: {}.freeze, group: [].freeze }.freeze
+
+    attr_reader :klass, :values
+
+    def initialize(klass, values = EMPTY)
+      @klass = klass
+      @values = values
+      @result = nil
+    end
+
+    # Builders
+
+    def where(attrs = :chain)
+      return WhereChain.new(self) if attrs == :chain
+      raise ArgumentError, "where takes a Hash, got #{attrs.class}" unless attrs.is_a?(Hash)
+
+      add_clause(Conditions.equality(attrs))
+    end
+
+    def or(other)
+      clauses = [Conditions.combine(values[:where]), Conditions.combine(other.values[:where])]
+
+      with(where: [{ "$or" => clauses }])
+    end
+
+    def order(*args)
+      sort = args.each_with_object(values[:order].dup) do |arg, out|
+        arg.is_a?(Hash) ? arg.each { |f, dir| out[f.to_s] = direction(dir) } : out[arg.to_s] = 1
+      end
+
+      with(order: sort)
+    end
+
+    def limit(count)
+      with(limit: count)
+    end
+
+    def offset(count)
+      with(offset: count)
+    end
+
+    def page(number)
+      with(page: number.to_i)
+    end
+
+    def per(size)
+      with(per: size.to_i)
+    end
+
+    def after(token)
+      with(after: token)
+    end
+
+    def group(*fields)
+      with(group: fields.flatten.map(&:to_s))
+    end
+
+    # The find body: JSON only, the daemon decides what it means.
+    def to_query
+      query = {}
+      where = Conditions.combine(values[:where])
+
+      query["where"] = where unless where.empty?
+      query["sort"] = values[:order] unless values[:order].empty?
+      query.merge!(window)
+      query["after"] = values[:after] if values[:after]
+
+      query
+    end
+
+    # Loaders
+
+    def load
+      return self if loaded?
+
+      @result = klass.ndb_find(to_query)
+      @records = @result.rows.map { |row| klass.instantiate(row) }
+
+      self
+    end
+
+    def loaded?
+      !@result.nil?
+    end
+
+    def records
+      load
+
+      @records
+    end
+
+    def to_a
+      records.dup
+    end
+
+    def each(&)
+      records.each(&)
+    end
+
+    def length
+      records.length
+    end
+
+    def next_cursor
+      load
+
+      @result.next_cursor
+    end
+
+    def first(count = nil)
+      return count ? records.first(count) : records.first if loaded?
+
+      found = limit(count || 1).to_a
+
+      count ? found : found.first
+    end
+
+    def find(*ids)
+      ids = ids.flatten
+
+      return find_one(ids.first) if ids.size == 1 && !ids.first.is_a?(Array)
+
+      found = where(klass.primary_key => ids).to_a
+      raise NotFound.new("Couldn't find all #{klass.name} with ids #{ids}", code: "not_found") if found.size != ids.size
+
+      found
+    end
+
+    def find_by(attrs)
+      where(attrs).first
+    end
+
+    def find_by!(attrs)
+      find_by(attrs) || raise(NotFound.new("Couldn't find #{klass.name}", code: "not_found"))
+    end
+
+    def exists?
+      klass.ndb_find(to_query.slice("where").merge("limit" => 1, "fields" => [klass.primary_key])).rows.any?
+    end
+
+    def pluck(*fields)
+      raise ArgumentError, "pluck needs at least one field" if fields.empty?
+
+      names = fields.map(&:to_s)
+      rows = klass.ndb_find(to_query.merge("fields" => names)).rows
+
+      rows.map do |row|
+        values = names.map { |name| klass.attribute_types[name].cast(row.dig(*name.split("."))) }
+
+        values.size == 1 ? values.first : values
+      end
+    end
+
+    def count
+      return calculate(:count) unless values[:group].empty?
+
+      klass.ndb_find(to_query.slice("where").merge("count" => true, "limit" => 1, "fields" => [klass.primary_key])).total
+    end
+
+    # Pagination
+
+    def find_each(batch_size: BATCH_SIZE, &block)
+      return enum_for(:find_each, batch_size: batch_size) unless block
+
+      in_batches(of: batch_size) { |batch| batch.each(&block) }
+    end
+
+    # Yields loaded relations, one per cursor page.
+    def in_batches(of: BATCH_SIZE)
+      return enum_for(:in_batches, of: of) unless block_given?
+
+      cursor = values[:after]
+
+      loop do
+        batch = with(limit: of, after: cursor, page: nil, offset: nil).load
+
+        break if batch.records.empty?
+
+        yield batch
+
+        cursor = batch.next_cursor
+
+        break unless cursor
+      end
+    end
+
+    # Aggregates
+
+    def sum(field) = calculate(:sum, field)
+    def average(field) = calculate(:avg, field)
+    def minimum(field) = calculate(:min, field)
+    def maximum(field) = calculate(:max, field)
+
+    def aggregate_body(op, field = nil)
+      stages = []
+      where = Conditions.combine(values[:where])
+
+      stages << { "$match" => where } unless where.empty?
+      stages << { "$group" => { "_id" => group_id, op.to_s => accumulator(op, field) } }
+
+      { "pipeline" => stages }
+    end
+
+    protected
+
+    def with(changes)
+      self.class.new(klass, values.merge(changes))
+    end
+
+    private
+
+    def add_clause(clause)
+      with(where: values[:where] + [clause])
+    end
+
+    def direction(value)
+      %w[desc -1].include?(value.to_s.downcase) ? -1 : 1
+    end
+
+    def window
+      if values[:page]
+        per = values[:per] || DEFAULT_PER_PAGE
+
+        return { "skip" => (values[:page] - 1) * per, "limit" => per }
+      end
+
+      out = {}
+      out["skip"] = values[:offset] if values[:offset]
+      out["limit"] = values[:limit] if values[:limit]
+
+      out
+    end
+
+    def find_one(id)
+      where(klass.primary_key => id).first ||
+        raise(NotFound.new("Couldn't find #{klass.name} with #{klass.primary_key}=#{id}", code: "not_found"))
+    end
+
+    def group_id
+      fields = values[:group]
+
+      return nil if fields.empty?
+      return "$#{fields.first}" if fields.one?
+
+      fields.to_h { |f| [f, "$#{f}"] }
+    end
+
+    def accumulator(op, field)
+      op == :count ? { "$count" => {} } : { "$#{op}" => "$#{field}" }
+    end
+
+    def calculate(op, field = nil)
+      key = op.to_s
+      rows = klass.ndb_aggregate(aggregate_body(op, field)).rows
+
+      return rows.to_h { |row| [group_key(row["_id"]), row[key]] } unless values[:group].empty?
+
+      rows.first&.fetch(key) || (op == :sum ? 0 : nil)
+    end
+
+    def group_key(id)
+      values[:group].one? ? id : values[:group].map { |f| id[f] }
+    end
+
+    # where.not(...)
+    class WhereChain
+      def initialize(relation)
+        @relation = relation
+      end
+
+      def not(attrs)
+        @relation.send(:add_clause, Conditions.negation(attrs))
+      end
+    end
+  end
+end

data/lib/ndb/rails/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Ndb
+  module Rails
+    VERSION = "0.1.0"
+  end
+end

data/lib/ndb/rails.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+require "active_model"
+require "active_support/all"
+require "ndb"
+
+require "ndb/rails/version"
+require "ndb/rails/errors"
+require "ndb/rails/configuration"
+require "ndb/rails/relation"
+require "ndb/rails/blob"
+require "ndb/rails/persistence"
+require "ndb/rails/record"
+
+module Ndb
+  module Rails
+    class << self
+      def configuration
+        @configuration ||= Configuration.new
+      end
+
+      def configure
+        yield configuration
+
+        @client = nil
+      end
+
+      def client
+        @client ||= Client.new(
+          url: configuration.url,
+          socket: configuration.socket,
+          token: configuration.token,
+          timeout: configuration.timeout,
+          pool_size: configuration.pool_size,
+          ready_timeout: configuration.ready_timeout
+        )
+      end
+    end
+  end
+end
+
+require "ndb/rails/railtie" if defined?(::Rails::Railtie)

data/lib/ndb-rails.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+require "ndb/rails"

metadata

@@ -0,0 +1,99 @@
+--- !ruby/object:Gem::Specification
+name: ndb-rails
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- thadeu
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: activemodel
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.1'
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.1'
+- !ruby/object:Gem::Dependency
+  name: ndb
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+description: Ndb::Record, a lazy Relation, validations, blobs and a Rails railtie
+  on top of the ndb client gem. The gem only builds JSON; the daemon decides every
+  query semantic.
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE
+- README.md
+- lib/ndb-rails.rb
+- lib/ndb/rails.rb
+- lib/ndb/rails/blob.rb
+- lib/ndb/rails/configuration.rb
+- lib/ndb/rails/errors.rb
+- lib/ndb/rails/persistence.rb
+- lib/ndb/rails/railtie.rb
+- lib/ndb/rails/record.rb
+- lib/ndb/rails/relation.rb
+- lib/ndb/rails/version.rb
+homepage: https://github.com/thadeu/ndb
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/thadeu/ndb
+  source_code_uri: https://github.com/thadeu/ndb/tree/main/sdk/rails
+  changelog_uri: https://github.com/thadeu/ndb/blob/main/CHANGELOG.md
+  bug_tracker_uri: https://github.com/thadeu/ndb/issues
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.2'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.9
+specification_version: 4
+summary: ActiveModel layer for the ndb NDJSON query daemon
+test_files: []