ndb 0.4.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 440cb991f003e2ce9f918da9d9f40fe3ac07e77ca117ba1b3d7dbd801d6b5866
-  data.tar.gz: 8d48e93a33a65dc8df214ff57ac8ef77d0c6edbb5c9bfa12cb364400ab2492d5
+  metadata.gz: 3cee02332a613d6dcb575f559504c2efde35cece0259791103d63113fa485ac1
+  data.tar.gz: 7a81d66e1e69c9208d24556c59e5249e413a9e71d45e774b581a05bca237df22
 SHA512:
-  metadata.gz: 2375c7fdd4cca6e943e6a5c27e192d4e6ed3dda62722f935bd4ffaa0c606fa2d973101e11bc1c19907dce0c0068366c2cb5cefba0022086e0dd85232cb78eaf8
-  data.tar.gz: d382affec84704deee77750b458da1b524f6805d08ebf3f38b3411c5a7b64b002eb4256ebe94f99243253e2ecb13f438207317cfc419aeecce12115b7ba1a350
+  metadata.gz: 29f3541621bd7f13917db6d2ccff09d5b400728ac14d9758df26dea0df529c43db95e0420a7d3977cfe2f3872d54be246ca18926b8cc3cbafd18ca3c2ef17945
+  data.tar.gz: e6904af991afc68bf5a131d299fe789c3b07225ba87ae1175d43e9ee468104b2f4a74e6e8e499d064d97ce56f907f253dc1ea641291d52691ec2021db09f6eae

data/README.md

@@ -4,9 +4,14 @@
 
 Plain Ruby client for [ndb](https://github.com/thadeu/ndb), an NDJSON query daemon. It talks HTTP over a unix socket or TCP, with a connection pool and keep-alive. It uses the Ruby standard library only: **no runtime dependencies**. The gem only builds JSON. The daemon decides what every query means.
 
+It has two layers:
+
+- `Ndb.connect` and `Ndb::Collection`: a collection with a query builder (`query`, `sort`, `limit`, `each`). Start here.
+- `Ndb::Client`: the low-level layer, one method per HTTP route. `Ndb.connect` returns one.
+
 Building a Rails app with models? Use [`ndb-rails`](https://github.com/thadeu/ndb/tree/main/sdk/rails) on top of this gem.
 
-Docs: [client guide](https://github.com/thadeu/ndb/blob/main/docs/dev/clients.md), [query language](https://github.com/thadeu/ndb/blob/main/docs/dev/query.md), [HTTP API](https://github.com/thadeu/ndb/blob/main/docs/dev/http-api.md).
+Docs: [client guide](https://github.com/thadeu/ndb/blob/main/docs/dev/clients.md), [query language](https://github.com/thadeu/ndb/blob/main/docs/dev/query.md), [HTTP API](https://github.com/thadeu/ndb/blob/main/docs/dev/http.md).
 
 ## Install
 
@@ -29,13 +34,14 @@ require "ndb"
 ## Connect
 
 ```ruby
-ndb = Ndb::Client.new(socket: "/run/ndb/ndb.sock")                       # unix socket
-ndb = Ndb::Client.new(url: "unix:///run/ndb/ndb.sock")                   # same, as a URL
-ndb = Ndb::Client.new(url: "tcp://10.0.0.5:7070", token: "secret")       # TCP with a bearer token
-ndb = Ndb::Client.new                                                    # reads NDB_URL and NDB_TOKEN
+ndb = Ndb.connect("unix:///run/ndb/ndb.sock")                 # unix socket
+ndb = Ndb.connect("tcp://10.0.0.5:7070", token: "secret")     # TCP with a bearer token
+ndb = Ndb.connect                                             # reads NDB_URL and NDB_TOKEN
+
+ndb.health   # => true when the daemon is ready (GET /_health)
 ```
 
-All options:
+`Ndb.connect(addr, **opts)` returns an `Ndb::Client`. It is the same as `Ndb::Client.new(url: addr, **opts)`. All options:
 
 ```ruby
 Ndb::Client.new(
@@ -48,130 +54,166 @@ Ndb::Client.new(
 )
 ```
 
-- The address is, in order: `url:`, `socket:`, env `NDB_URL`, then the default socket `/var/run/ndb/ndb.sock`.
+- The address is, in order: `url:` (the `addr` of `Ndb.connect`), `socket:`, env `NDB_URL`, then the default socket `/var/run/ndb/ndb.sock`.
 - The token is `token:`, or env `NDB_TOKEN`. It goes only in the `Authorization` header. A unix socket does not need it.
 - A bad address raises `ArgumentError` at once.
 - `ready_timeout` is how long a call retries while the daemon is not ready (`not_ready`, or the socket is missing or refuses). It uses exponential backoff (50 ms up to 1 s) with jitter. A write is retried only when the daemon certainly did not process it. A read is also retried after a dropped connection. A timeout is never retried. When the deadline passes, the last error is raised.
+- `health` does not wait: a daemon that is recovering raises `Ndb::NotReady`.
 - A reused keep-alive connection that the server closed is sent again only for a read. A write is never sent twice.
 - The pool is thread-safe and drops its connections after a `fork`.
 
 ## Collections
 
 ```ruby
-ndb.ensure_collection("calls",
-                      indexes: ["status"],
-                      unique: ["caller"],
-                      time: "at")      # also: expires_field:, ttl:
+calls = ndb.collection("calls",
+                       indexes: ["status"],
+                       unique: ["caller"],
+                       time: "at")      # also: expires_field:, ttl:
 
-ndb.collections   # => [{"name" => "calls", ...}]
+ndb.collections   # => [{"name" => "calls", ...}] (GET /_collections)
 ```
 
-Every collection has the primary key `_id`. You can send your own (any non-empty string), or leave it out: the daemon makes a UUIDv7 (`Ndb.new_id` makes the same kind of id on the client). For a natural key such as an email, keep `_id` and declare that field in `unique:`.
+- `collection` sends nothing. The first call on the collection declares it (`PUT /calls`) with this definition, once per object. A different definition for an existing collection raises `Ndb::CatalogConflict` on that call.
+- A collection declared in `ndb.yml` needs no definition: `ndb.collection("app_logs")`.
+- A name that ends with `*` is a collection pattern: `ndb.collection("logs-2026-*")` reads every collection whose name starts with `logs-2026-`. It declares nothing, and a write raises `Ndb::Error` with code `pattern_read_only`. Each row has `"_coll"`, the name of its collection.
 
-`ensure_collection` is idempotent. A different definition for an existing collection raises `Ndb::CatalogConflict`.
+Every collection has the primary key `_id`. You can send your own (any non-empty string), or leave it out: the daemon makes a UUIDv7 (`Ndb.new_id` makes the same kind of id on the client). For a natural key such as an email, keep `_id` and declare that field in `unique:`.
 
 ## Insert, update, delete
 
-Every document has a version. A new document has version 1, and each change adds 1.
+Every document has a version, `_v`. A new document has version 1, and each change adds 1.
 
 ```ruby
-results = ndb.insert("calls", [
-  { _id: "c1", status: "answered", caller: "a", duration: 42 },
-  { _id: "c2", status: "failed", caller: "b", duration: 5 }
+calls.insert(_id: "c1", status: "answered", caller: "a", duration: 42)   # one document (POST /calls/_doc)
+# => {"_id" => "c1", "_v" => 1}
+
+calls.insert([                                                             # many in one transaction (POST /calls/_bulk)
+  { _id: "c2", status: "failed", caller: "b", duration: 5 },
+  { _id: "c3", status: "failed", caller: "c", duration: 7 }
 ])
-results   # => [{"_id" => "c1", "v" => 1}, {"_id" => "c2", "v" => 1}]
+# => [{"_id" => "c2", "_v" => 1}, {"_id" => "c3", "_v" => 1}]
+
+calls.upsert(_id: "c2", status: "answered")              # create by _id, or merge into the stored document
+calls.upsert_all([{ _id: "c2", duration: 9 }, { _id: "c4", status: "new" }])   # one transaction
 
-ndb.get("calls", "c1")   # => {"_id" => "c1", ..., "_v" => 1}, or nil
+calls.get("c1")   # => {"_id" => "c1", ..., "_v" => 1}, or nil (GET /calls/_doc/c1)
 
-# Atomic update by _id (a String) or by filter (a Hash): the target, then the write operators.
-ndb.update("calls", "c1", { "$set" => { status: "done" }, "$unset" => { caller: true } })
+# Atomic update by _id (POST /calls/_update/c1): the id, then the write operators.
+calls.update("c1", { "$set" => { status: "done" }, "$unset" => { caller: true } })
 # => {"matched" => 1, "modified" => 1, "doc" => {"_id" => "c1", ..., "_v" => 2}}
 
-ndb.update("calls", { _id: "c1", _v: 2 }, { "$set" => { duration: 50 } })   # only if the version is still 2
-ndb.update("stock", { _id: "a", qty: { "$gte" => 1 } }, { "$inc" => { qty: -1 } })   # guarded decrement
+calls.update("c1", { "$set" => { duration: 50 } }, guard: { _v: 2 })   # only if the version is still 2
+stock.update("a", { "$inc" => { qty: -1 } }, guard: { qty: 1.. })      # guarded decrement
+counters.update("hits", { "$inc" => { n: 1 } }, upsert: true)          # created with this _id when missing
 
-ndb.update("calls", { status: "failed" }, { "$set" => { status: "retry" } }, many: true)   # => {"matched" => 3, "modified" => 3}
-ndb.update("counters", { name: "hits" }, { "$inc" => { n: 1 } }, upsert: true)
+# Atomic update by query (POST /calls/_update_by_query): the first match, or every match with many.
+calls.update_by({ status: "failed" }, { "$set" => { status: "retry" } }, many: true)   # => {"matched" => 2, "modified" => 2}
+calls.update_by({ caller: "z" }, { "$inc" => { duration: 1 } }, upsert: true)
 
-ndb.delete("calls", "c2")                                # by _id => {"deleted" => 1}
-ndb.delete("calls", { status: "failed" }, many: true)
+calls.delete("c2")                                  # by _id => {"deleted" => 1} (DELETE /calls/_doc/c2)
+calls.delete_by({ status: "retry" }, many: true)    # by query (POST /calls/_delete_by_query)
 ```
 
 - Operators: `$set`, `$setOnInsert` (like `$set`, but only when an upsert creates the document), `$unset`, `$inc`, `$push` (with `{ "$each" => [...] }`), `$pull`, `$min`, `$max`. Field names can be dot paths. A key of the operators without `$` raises `ArgumentError` before anything is sent.
-- Nothing matched is not an error: `matched` is 0. A stale `_v` is a filter that matches nothing, so read again and retry. `modified` is 0 when the operators leave the document as it is.
-- `many: true` needs `_id` or an indexed field in the filter (`Ndb::UnindexedMany`), and does not apply to an `_id` String. `upsert: true` creates the document from the equality fields of the filter. The answer has `doc` only for one document.
+- Nothing matched is not an error: `matched` is 0. A stale `_v` guard matches nothing, so read again and retry. `modified` is 0 when the operators leave the document as it is.
+- `update` takes `guard:`, `upsert:` (creates the document with that `_id`) and `expire_in:`. `update_by` takes `many:`, `upsert:` (creates the document from the equality fields of the query) and `expire_in:`. `delete_by` takes `many:`. The answer has `doc` only for one document.
+- The query and the guard take the Ruby forms of `query` (below): an Array is `$in`, a Range is `$gte`/`$lte`, `nil` is `$exists: false`.
+- `many: true` needs `_id` or an indexed field in the query (`Ndb::UnindexedMany`). An empty query raises `Ndb::EmptyFilter`.
 - The contract errors are `Ndb::UnknownOperator`, `NoUpdate`, `ConflictingUpdate`, `InvalidField`, `UnindexedMany` and `EmptyFilter`, all under `Ndb::InvalidUpdate`.
-- A duplicate `_id` raises `Ndb::UniqueViolation` (code `duplicate_id`). A broken unique field raises it too (code `unique_violation`).
+- A duplicate `_id` raises `Ndb::UniqueViolation` (code `duplicate_id`), and a many-document insert writes nothing. A broken unique field raises it too (code `unique_violation`).
 
 ### Truncate and drop
 
 ```ruby
-ndb.truncate("calls")   # every document goes; the definition and indexes stay => {"collection" => "calls", "truncated" => true}
-ndb.drop("calls")       # the collection, its definition and its watch consumers go
+calls.truncate   # every document goes; the definition and indexes stay => {"collection" => "calls", "truncated" => true}
+calls.drop       # the collection, its definition and its watch consumers go
 ```
 
 - Both take the same time for 10 or 10 million documents. Watch consumers get one event with `"op" => "truncate"` (no `_id`), then the new changes.
-- A direct-mode collection raises `Ndb::ReadOnlyCollection`. A collection declared in `ndb.yml` comes back, empty, when the daemon restarts after a drop.
+- After `drop`, the next call on the collection declares it again. A collection declared in `ndb.yml` comes back, empty, when the daemon restarts.
+- A direct-mode collection raises `Ndb::ReadOnlyCollection`.
 
 ## Expiry (TTL like Redis)
 
 The collection needs `expires_field`.
 
 ```ruby
-ndb.ensure_collection("sessions", { expires_field: "exp" })
+sessions = ndb.collection("sessions", expires_field: "exp")
 
-ndb.insert("sessions", [{ _id: "s1" }], expire_in: 3600)  # seconds, or "1h" / "30m" / "1d"
-ndb.update("sessions", "s1", { "$set" => { user: "u" } }, expire_in: "30m")
+sessions.insert({ _id: "s1" }, expire_in: 3600)                 # seconds, or "1h" / "30m" / "1d"
+sessions.update("s1", { "$set" => { user: "u" } }, expire_in: "30m")
 
-ndb.expire("sessions", "s1", in: 1800)                     # => {"_id", "v", "expires_at"}
-ndb.expire("sessions", "s1", at: Time.now + 7200)
-ndb.ttl("sessions", "s1")                                  # => 7199, -1 (never expires), or nil (missing or expired)
-ndb.persist("sessions", "s1")                              # => {"_id", "v"}
+sessions.expire("s1", in: 1800)                                  # => {"_id", "_v", "expires_at"}
+sessions.expire("s1", at: Time.now + 7200)
+sessions.ttl("s1")                                               # => 7199, -1 (never expires), or nil (missing or expired)
+sessions.persist("s1")                                           # => {"_id", "_v"}
 ```
 
 - `expire_in` wins over a value of `expires_field` in the same call. `ttl` rounds up to whole seconds.
 - `expire` and `persist` raise `Ndb::NotFound` for a missing or expired document. `Ndb::TtlNotEnabled` means no `expires_field`; `Ndb::TtlDerived` means the expiry comes from `ttl` + `time`.
 - Disk space of expired documents returns after compaction, and fully only with `wal_retention`; see [Freeing disk](https://github.com/thadeu/ndb/blob/main/docs/dev/configuration.md#freeing-disk).
 
-## Find
+## Query
 
-`find` returns a `Ndb::Page` with `rows` (an array of Hash), `next_cursor`, `plan`, `examined` and `total`.
+A query is immutable: every step returns a new one. The request (`POST /{coll}/_search`) goes out when you call `all`, `first`, `page`, `count` or `each`.
 
 ```ruby
-page = ndb.find("calls",
-                where: { status: "answered", duration: { "$gte" => 30, "$lte" => 60 } },
-                sort: { at: -1 },
-                fields: %w[_id status],
-                limit: 50,
-                skip: 0)
-
-page.rows          # => [{"_id" => "c1", "status" => "answered"}, ...]
-page.plan          # "id", "index:status", "scan" or "snapshot"
-page.examined      # rows the daemon looked at
+calls.query(status: "answered").all
+calls.query(status: %w[answered failed])            # $in
+calls.query(duration: 30..60)                       # $gte and $lte
+calls.query(duration: 30...60)                      # $gte and $lt
+calls.query(duration: nil)                          # $exists: false
+calls.query(at: Time.now - 86_400..)
+calls.query("agent.id" => 7)                        # dot path
+calls.query(duration: { "$gt" => 10 })              # any daemon operator, as a Hash
+calls.query("$or" => [{ status: "failed" }, { duration: { "$gt" => 50 } }])
+
+calls.query(status: "answered")
+     .query(caller: %w[a b])                        # ANDed
+     .sort(at: :desc, _id: :asc)
+     .fields(:_id, :status)
+     .limit(50)
+     .skip(100)
+     .all                                           # => [{"_id" => "c1", "status" => "answered"}, ...]
+
+calls.query(status: "answered").first               # the first row, or nil
 ```
 
-The body of `find` is the JSON of the [query language](https://github.com/thadeu/ndb/blob/main/docs/dev/query.md): `where`, `sort`, `fields`, `limit`, `skip`, `after`, `count`, `explain`, `versions`. Symbol keys are fine.
+- `query` turns Ruby values into operators: an Array is `$in`, a Range is `$gte` and `$lte` (or `$lt`), `nil` is `$exists: false`, a `Time` is an RFC 3339 string, a `Regexp` is `$regex`. A key that starts with `$` is passed as is.
+- `sort(at: -1, _id: 1)`: `1`, `:asc` ascending, `-1`, `:desc` descending. The key order is the sort order. The id is always the last key.
+- `fields` returns only these fields.
+- See the [query language](https://github.com/thadeu/ndb/blob/main/docs/dev/query.md) for every operator.
+
+### A query built elsewhere
+
+A query from a search form or a saved search is a `raw_query`. Its Hash is sent as it is (only Times and Regexps are converted), ANDed with the other conditions:
 
-In `find`, `where` is sent as it is: write the daemon operators yourself. (Only `watch` turns Ruby values such as Range and Array into operators, see below.)
+```ruby
+calls.raw_query("status" => "answered", "at" => { "$gte" => "now-7d" }).limit(50).all
+```
+
+For a whole search body (`query`, `sort`, `fields`, `limit`, `skip`, `after`, `count`, `explain`), use `search`. It returns one `Ndb::Page`:
 
 ```ruby
-ndb.find("calls", where: { status: { "$in" => %w[answered failed] } })
-ndb.find("calls", where: { duration: { "$gte" => 30, "$lte" => 60 } })
-ndb.find("calls", where: { "$or" => [{ status: "failed" }, { duration: { "$gt" => 50 } }] })
-ndb.find("calls", where: { "agent.id" => 7 })   # dot path
+page = calls.search(query: { status: "answered" }, sort: { at: -1 }, limit: 50, count: true)
+
+page.rows       # => [{"_id" => "c1", ...}, ...]
+page.next       # the cursor of the next page, nil on the last one
+page.total      # with count: true
+page.plan       # "id", "index:status", "scan" or "snapshot"
+page.examined   # rows the daemon looked at
 ```
 
-- `sort: { at: -1, _id: 1 }`: `1` ascending, `-1` descending. The key order is the sort order. The id is always the last key.
-- `versions: true` adds the field `_v` (the document version) to each row.
+`query(...).page` returns the same `Ndb::Page` for a built query (`page(count: true)` fills `total`).
 
 ### Regular expressions
 
-A Ruby `Regexp` is accepted anywhere a value is: a field value, `"$regex" => /x/`, inside `$not`, `$or`, `$and`, in `find`, `aggregate` (`$match`) and in the `where:` of `watch`.
+A Ruby `Regexp` is accepted anywhere a value is: a field value, `"$regex" => /x/`, inside `$not`, `$or`, `$and`, in `query`, `search`, `aggregate` (`$match`) and in the `query:` of `watch`.
 
 ```ruby
-ndb.find("calls", where: { name: /^alpha/i })                    # i: ignore case
-ndb.find("calls", where: { name: { "$regex" => /bet$/ } })
-ndb.find("calls", where: { "$or" => [{ name: /^a/ }, { name: /^b/ }] })
+calls.query(name: /^alpha/i)                                     # i: ignore case
+calls.query(name: { "$regex" => /bet$/ })
+calls.query("$or" => [{ name: /^a/ }, { name: /^b/ }])
 ```
 
 The daemon uses RE2. The mapping to `$options` is:
@@ -185,56 +227,67 @@ The daemon uses RE2. The mapping to `$options` is:
 
 Ruby features that RE2 does not have raise `Ndb::InvalidQuery` (code `invalid_query`) **before** any request: look-around (`(?=`, `(?!`, `(?<=`, `(?<!`), backreferences (`\1` to `\9`, `\k<name>`), atomic groups (`(?>`), possessive quantifiers (`a++`), `\h`, `\G`, `\R` and `\X`.
 
-A String pattern is sent unchanged and has no `m`: `where: { name: { "$regex" => "^abc" } }`. A pattern anchored with `^` and a literal start, without `i` or `m`, uses the index on the field (`page.plan` is `index:name`). A Regexp always sends `m`, so it is a scan filter, not an index range. For a prefix search that uses the index, send the String form.
+A String pattern is sent unchanged and has no `m`: `query(name: { "$regex" => "^abc" })`. A pattern anchored with `^` and a literal start, without `i` or `m`, uses the index on the field (`page.plan` is `index:name`). A Regexp always sends `m`, so it is a scan filter, not an index range. For a prefix search that uses the index, send the String form.
 
 ### Count
 
 ```ruby
-page = ndb.find("calls", where: { status: "answered" }, count: true, limit: 1)
-page.total   # how many documents match, whatever the limit
+calls.count(status: "answered")            # POST /calls/_count
+calls.query(status: "answered").count      # the same
+calls.count                                # every document
 ```
 
+`limit`, `skip` and `sort` do not change the count.
+
 ### Cursor pagination
 
+Offset pages get slower as `skip` grows. Use a cursor for deep pages.
+
 ```ruby
-cursor = nil
+query = calls.query(status: "answered").sort(_id: 1).limit(100)
 
 loop do
-  page = ndb.find("calls", where: { status: "answered" }, sort: { _id: 1 }, limit: 100, after: cursor)
+  page = query.page
 
   page.rows.each { |row| puts row["_id"] }
 
-  cursor = page.next_cursor
+  break if page.next.nil?   # nil on the last page
 
-  break if cursor.nil?   # nil on the last page
+  query = query.after(page.next)
 end
 ```
 
-A cursor works only with the same `where` and `sort`. Another query shape raises `Ndb::InvalidCursor`.
+A cursor works only with the same query and `sort`. Another query shape raises `Ndb::InvalidCursor`.
+
+To walk every row, use `each`. It follows the cursor, with `limit` as the batch size. A query is `Enumerable`:
+
+```ruby
+calls.query(status: "answered").sort(_id: 1).limit(500).each { |row| puts row["_id"] }
+```
 
 ## Aggregate
 
 The pipeline needs exactly one `$group`. Accumulators: `$count`, `$sum`, `$avg`, `$min`, `$max`, `$first`, `$last`.
 
 ```ruby
-page = ndb.aggregate("calls",
-                     pipeline: [
-                       { "$match" => { "duration" => { "$gte" => 1 } } },
-                       { "$group" => {
-                         "_id" => "$status",
-                         "n" => { "$count" => {} },
-                         "avg" => { "$avg" => "$duration" }
-                       } },
-                       { "$sort" => { "n" => -1 } }
-                     ],
-                     limit: 100,
-                     count: true)
+page = calls.aggregate([
+                         { "$match" => { "duration" => { "$gte" => 1 } } },
+                         { "$group" => {
+                           "_id" => "$status",
+                           "n" => { "$count" => {} },
+                           "avg" => { "$avg" => "$duration" }
+                         } },
+                         { "$sort" => { "n" => -1 } }
+                       ],
+                       limit: 100,
+                       count: true)
 
 page.rows    # => [{"_id" => "answered", "n" => 2, "avg" => 35.5}, ...]
 page.total   # with count: true
+page.next    # the cursor of the next page, nil on the last one
 ```
 
-`$dateTrunc` units are `minute`, `hour`, `day`, `week` and `month`.
+Options: `limit:`, `skip:`, `after:`, `count:`. `$dateTrunc` units are `minute`, `hour`, `day`, `week` and `month`.
 
 ## Watch
 
@@ -242,11 +295,11 @@ A watch streams the changes of a collection to a named consumer: first the event
 
 ```ruby
 begin
-  ndb.watch("calls", "relay", where: { status: "failed" }) do |event|
-    # event has "pos", "op" ("put", "del" or "truncate"), "_id", "v" and "doc"
+  calls.watch("relay", query: { status: "failed" }) do |event|
+    # event has "pos", "op" ("put", "del" or "truncate"), "_id", "_v" and "doc"
     handle(event)
 
-    ndb.ack("calls", "relay", event["pos"])
+    calls.ack(event, "relay")
   end
 rescue Ndb::WatchLagged
   retry # the consumer was too slow. watch continues after the last ack.
@@ -255,10 +308,11 @@ rescue Ndb::HistoryTruncated
 end
 ```
 
-- `where:` takes the same conditions as the models of `ndb-rails`: equality, an Array (`$in`), a Range, a Regexp (see Regular expressions), `nil` (`$exists: false`) and `$` operators as keys. Only matching events come. A delete event has no `"doc"`; it is sent only if this stream saw an earlier matching put of that id.
+- `query:` takes the same conditions as `query`: equality, an Array (`$in`), a Range, a Regexp, `nil` (`$exists: false`) and `$` operators as keys. Only matching events come. A delete event has no `"doc"`; it is sent only if this stream saw an earlier matching put of that id.
 - `from: "now"` starts a new consumer at the end. `limit: n` ends the stream after `n` events.
-- Without a block, `watch` returns an Enumerator: `ndb.watch("calls", "relay", limit: 10).to_a`.
-- Delivery is at least once. After a restart an event can come again, so make the work safe to repeat. `ack` stores the position of an event: the next `watch` with that consumer name starts after it.
+- Without a block, `watch` returns an Enumerator: `calls.watch("relay", limit: 10).to_a`.
+- `ack(event, consumer)` stores the position of the event (`POST /calls/_ack`): the next `watch` with that consumer name starts after it. The event Hash has no consumer name, so pass it.
+- Delivery is at least once. After a restart an event can come again, so make the work safe to repeat.
 - `HistoryTruncated` means the WAL files for the position were deleted (`wal_retention`).
 - A watch holds one connection until it ends. It is retried only for the initial open, never in the middle of a stream.
 
@@ -267,17 +321,68 @@ end
 Large values live in the blob store. A document keeps only a reference.
 
 ```ruby
-stored = ndb.put_blob(File.open("call.wav"), "audio/wav")   # an IO, streamed in chunks
+stored = ndb.put_blob(File.open("call.wav"), "audio/wav")   # an IO, streamed in chunks (POST /_blobs)
 stored   # => {"$blob" => "sha256:...", "size" => 123456, "type" => "audio/wav"}
 
-ndb.insert("calls", [{ _id: "c9", recording: stored }])
+calls.insert(_id: "c9", recording: stored)
 
-ndb.get_blob(stored["$blob"])                    # all the bytes
+ndb.get_blob(stored["$blob"])                    # all the bytes (GET /_blobs/{ref})
 ndb.get_blob(stored["$blob"], from: 0, to: 1023) # bytes 0..1023 (inclusive)
 ```
 
 The same content always gives the same reference. An IO upload is never retried.
 
+## Admin
+
+```ruby
+calls.stats                 # rows, indexes, files, bytes (GET /calls/_stats)
+calls.compact               # merge every segment now (POST /calls/_compact)
+calls.ensure_index("caller") # built in the background (PUT /calls/_indexes/caller) => the indexed fields
+calls.drop_index("caller")
+
+ndb.collections             # one definition per collection (GET /_collections)
+ndb.stats                   # every collection, totals, version, uptime (GET /_stats)
+ndb.size                    # disk use per collection and part (GET /_size)
+```
+
+## The low-level client
+
+`Ndb::Client` has one method per route. The collection methods above call it. Use it when the collection name is only known at run time, or to send a body built elsewhere. The first argument is the collection name.
+
+```ruby
+ndb.ensure_collection("calls", indexes: ["status"], time: "at")    # PUT /calls
+
+ndb.insert("calls", { _id: "c1", status: "answered" })             # POST /calls/_doc => {"_id", "_v"}
+ndb.bulk("calls", [{ _id: "c2" }, { _id: "c3" }], expire_in: "1h") # POST /calls/_bulk => the results
+ndb.get("calls", "c1")                                              # GET /calls/_doc/c1 => the document, or nil
+
+ndb.update("calls", "c1", { "$set" => { status: "done" } }, guard: { _v: 1 })                 # POST /calls/_update/c1
+ndb.update_by("calls", { status: "failed" }, { "$set" => { status: "retry" } }, many: true)    # POST /calls/_update_by_query
+ndb.delete("calls", "c2")                                                                      # DELETE /calls/_doc/c2
+ndb.delete_by("calls", { status: "retry" }, many: true)                                        # POST /calls/_delete_by_query
+
+page = ndb.search("calls", query: { status: "answered" }, sort: { at: -1 }, limit: 50)   # POST /calls/_search => an Ndb::Page
+ndb.count("calls", { status: "answered" })                                                # POST /calls/_count => an Integer
+ndb.aggregate("calls", pipeline: [{ "$group" => { "_id" => "$status", "n" => { "$count" => {} } } }])   # POST /calls/_aggregate
+
+ndb.expire("calls", "c1", in: "30m")    # POST /calls/_expire/c1
+ndb.persist("calls", "c1")              # POST /calls/_persist/c1
+ndb.ttl("calls", "c1")                  # GET /calls/_ttl/c1
+
+ndb.watch("calls", "relay", query: { status: "failed" }) { |event| ndb.ack("calls", "relay", event["pos"]) }   # POST /calls/_watch, /calls/_ack
+
+ndb.truncate("calls")                   # POST /calls/_truncate?confirm=calls
+ndb.drop("calls")                       # DELETE /calls?confirm=calls
+ndb.ensure_index("calls", "caller")     # PUT /calls/_indexes/caller
+ndb.drop_index("calls", "caller")       # DELETE /calls/_indexes/caller
+ndb.collection_stats("calls")           # GET /calls/_stats
+ndb.compact("calls")                    # POST /calls/_compact
+```
+
+- `search`, `count` and `aggregate` send the body as it is: write the daemon operators yourself. Only a Regexp is turned into `$regex`.
+- `update`, `update_by`, `delete_by` and `watch` take the Ruby forms of `query` (Array, Range, `nil`, Time, Regexp) in the query and the guard.
+- `ndb.collection(name, definition)` returns the `Ndb::Collection` of the sections above.
+
 ## Errors
 
 All errors inherit from `Ndb::Error`, which has `code` (the daemon code) and `status` (the HTTP status).
@@ -300,11 +405,11 @@ All errors inherit from `Ndb::Error`, which has `code` (the daemon code) and `st
 | `TtlNotEnabled`, `TtlDerived` | `ttl_not_enabled`, `ttl_derived` |
 | `ConnectionError` | cannot reach the daemon. Subclasses: `ConnectError` (nothing was sent), `TimeoutError` |
 
-Any other code is a plain `Ndb::Error`. A response that is not a daemon error has the code `http_status`.
+Any other code (for example `unknown_route`, `pattern_read_only`) is a plain `Ndb::Error`. A response that is not a daemon error has the code `http_status`.
 
 ```ruby
 begin
-  ndb.insert("calls", [{ _id: "c1" }])
+  calls.insert(_id: "c1")
 rescue Ndb::UniqueViolation => e
   e.code    # "duplicate_id"
   e.status  # 409
@@ -313,14 +418,14 @@ end
 
 ## Testing
 
-The tests of this gem build the daemon from the repository (`go build ./cmd/ndb`), start it on a short temp unix socket (macOS allows 104 bytes), wait for `GET /health` to return 200, and stop it at exit. To do the same by hand:
+The tests of this gem build the daemon from the repository (`go build ./cmd/ndb`), start it on a short temp unix socket (macOS allows 104 bytes), wait for `GET /_health` to return 200, and stop it at exit. To do the same by hand:
 
 ```sh
 ndb -config /nonexistent.yml -data-dir "$(mktemp -d)" -unix /tmp/ndb-test.sock
 ```
 
 ```ruby
-ndb = Ndb::Client.new(socket: "/tmp/ndb-test.sock")
+ndb = Ndb.connect("unix:///tmp/ndb-test.sock")
 ```
 
 Use a new collection name per test. Run the suite with `bundle exec rake test` in `sdk/ruby`.

data/lib/ndb/client.rb

@@ -5,10 +5,14 @@ require "uri"
 require "time"
 
 module Ndb
-  Page = Struct.new(:rows, :next_cursor, :plan, :examined, :total)
-
-  # The HTTP API of the daemon (plan §12.1). It builds JSON and reads
-  # NDJSON; it has no query semantics.
+  # One page of a search or an aggregate: the rows, the cursor of the next
+  # page (nil on the last one), the plan, how many rows were examined, and
+  # the total (with count: true).
+  Page = Data.define(:rows, :next, :plan, :examined, :total)
+
+  # The HTTP API of the daemon, one method per route. It builds JSON and
+  # reads the answers; it has no query semantics. Ndb::Collection is the
+  # friendlier layer on top of it.
   class Client
     DEFAULT_SOCKET = "/var/run/ndb/ndb.sock"
 
@@ -33,123 +37,153 @@ module Ndb
 
     def socket = @endpoint.path
 
-    def find(coll, body)
-      page(post(coll, "find", Conditions.regexps(body)))
+    # A collection (or a pattern such as "logs-2026-*") with the query
+    # builder. definition is what the first call declares (PUT /{coll}).
+    def collection(name, definition = {})
+      Collection.new(self, name, definition)
+    end
+
+    # POST /{coll}/_search: body is { "query", "sort", "fields", "limit",
+    # "skip", "after", "count", ... }. Returns a Page.
+    def search(coll, body = {})
+      page(post(coll, "_search", Conditions.regexps(body)))
+    end
+
+    # POST /{coll}/_count: how many documents match query.
+    def count(coll, query = nil)
+      body = query ? { query: Conditions.regexps(query) } : {}
+
+      json(post(coll, "_count", body)).fetch("count")
     end
 
+    # POST /{coll}/_aggregate: body is { "pipeline", "limit", "skip",
+    # "after", "count" }. Returns a Page.
     def aggregate(coll, body)
-      page(post(coll, "aggregate", Conditions.regexps(body)))
+      page(post(coll, "_aggregate", Conditions.regexps(body)))
     end
 
     # The document with this _id (and its "_v"), or nil.
     def get(coll, id)
-      json(request("GET", id_path(coll, id), nil))
+      json(request("GET", doc_path(coll, "_doc", id), nil))
     rescue NotFound
       nil
     end
 
-    # Each document gets a generated UUIDv7 _id unless it carries one.
+    # POST /{coll}/_doc: one document; it gets a generated UUIDv7 _id unless
+    # it carries one. Returns {"_id", "_v"}.
+    def insert(coll, doc)
+      json(post(coll, "_doc", doc))
+    end
+
+    # POST /{coll}/_bulk: many documents in one transaction, all or nothing.
     # expire_in (seconds, or "30m" / "1h" / "1d") sets the expires_field of
-    # every document to now + expire_in, in the same transaction.
-    def insert(coll, docs, expire_in: nil)
-      body = { insert: docs }
+    # every document to now + expire_in. Returns the results.
+    #
+    # upsert: true merges a document whose _id exists into the stored one
+    # (its fields replace the fields of the same name; the other fields
+    # stay), like doc_as_upsert in Elasticsearch, instead of duplicate_id.
+    def bulk(coll, docs, expire_in: nil, upsert: false)
+      body = { docs: docs }
       body[:expire_in] = expire_in if expire_in
+      body[:upsert] = true if upsert
 
-      json(post(coll, "insert", body)).fetch("results")
+      json(post(coll, "_bulk", body)).fetch("results")
     end
 
-    # Atomic update. A String is the _id of one document (POST /{_id}/update);
-    # a Hash is a filter (POST /update). The filter and the write operators go
-    # in one body; the daemon matches and writes in one step on the newest state, so
-    # concurrent updates never interleave.
+    # POST /{coll}/_update/{id}: applies the write operators to the document
+    # with this _id, in one atomic step. guard adds conditions on the same
+    # document ({ _v: 7 }); matched is 0 when they fail.
     #
-    #   client.update("stock", { sku: "a", qty: { "$gte" => 1 } }, { "$inc" => { qty: -1 } })
+    #   client.update("stock", "a", { "$inc" => { qty: -1 } }, guard: { qty: { "$gte" => 1 } })
     #   # => {"matched" => 1, "modified" => 1, "doc" => {...with "_v"}}
     #
-    # Operators: $set, $setOnInsert, $unset, $inc, $push, $pull, $min, $max (dot paths work).
-    # Nothing matched is not an error: {"matched" => 0, "modified" => 0}. A
-    # version guard is a filter: { _id: "x", _v: 7 }. many: true updates every
-    # match (no "doc" in the answer) and needs _id or an indexed field in the
-    # filter; it does not apply to an _id String. upsert: true creates the document from the equality fields
-    # of the filter when nothing matches. expire_in (seconds, or "30m") sets
-    # the expires_field.
-    def update(coll, id_or_filter, ops, many: false, upsert: false, expire_in: nil)
-      if id_or_filter.is_a?(String)
-        query = write_query(upsert: upsert, expire_in: expire_in)
-
-        return json(request("POST", "#{id_path(coll, id_or_filter)}/update#{query}", merge_update({}, ops)))
-      end
+    # Operators: $set, $setOnInsert, $unset, $inc, $push, $pull, $min, $max
+    # (dot paths work). upsert: true creates the document with this _id.
+    def update(coll, id, ops, guard: nil, upsert: false, expire_in: nil)
+      body = write_body(update: check_ops(ops), query: guard && Conditions.equality(guard), upsert: upsert, expire_in: expire_in)
+
+      json(request("POST", doc_path(coll, "_update", id), body))
+    end
 
-      query = write_query(many: many, upsert: upsert, expire_in: expire_in)
+    # POST /{coll}/_update_by_query: the atomic update by query. The daemon
+    # matches and writes in one step on the newest state, so concurrent
+    # updates never interleave. Without many it updates the first match;
+    # many: true updates every match and needs _id or an indexed field in
+    # the query. upsert: true creates the document from the equality fields
+    # of the query when nothing matches.
+    def update_by(coll, query, ops, many: false, upsert: false, expire_in: nil)
+      body = write_body(query: Conditions.equality(query), update: check_ops(ops), many: many, upsert: upsert, expire_in: expire_in)
 
-      json(post(coll, "update#{query}", merge_update(id_or_filter, ops)))
+      json(post(coll, "_update_by_query", body))
     end
 
-    # Deletes by _id (a String), or the documents that the filter matches (one,
-    # or every match with many: true, which needs _id or an indexed field).
-    # {"deleted" => n}
-    def delete(coll, id_or_filter, many: false)
-      return json(request("DELETE", id_path(coll, id_or_filter), nil)) if id_or_filter.is_a?(String)
+    # DELETE /{coll}/_doc/{id}: {"deleted" => 0 | 1}.
+    def delete(coll, id)
+      json(request("DELETE", doc_path(coll, "_doc", id), nil))
+    end
 
-      json(post(coll, "delete#{write_query(many: many)}", id_or_filter))
+    # POST /{coll}/_delete_by_query: the first match, or every match with
+    # many: true (same rule as update_by). {"deleted" => n}
+    def delete_by(coll, query, many: false)
+      json(post(coll, "_delete_by_query", write_body(query: Conditions.equality(query), many: many)))
     end
 
     # Makes a document expire, like EXPIRE in Redis: in: seconds or "30m", or
-    # at: a Time or an RFC 3339 String. Returns {"_id", "v", "expires_at"}.
+    # at: a Time or an RFC 3339 String. Returns {"_id", "_v", "expires_at"}.
     # Raises NotFound for a missing or expired document, TtlNotEnabled when
     # the collection has no expires_field, TtlDerived when the expiry comes
     # from ttl + time.
     def expire(coll, id, **time)
-      body = { _id: id }
+      body = {}
       body[:in] = time[:in] if time[:in]
       body[:at] = time[:at].respond_to?(:utc) ? time[:at].utc.iso8601(9) : time[:at] if time[:at]
 
-      json(post(coll, "expire", body))
+      json(request("POST", doc_path(coll, "_expire", id), body))
     end
 
     # Removes the expiry of a document, like PERSIST in Redis.
     def persist(coll, id)
-      json(post(coll, "persist", _id: id))
+      json(request("POST", doc_path(coll, "_persist", id), nil))
     end
 
     # The seconds left of a document (an Integer, rounded up), -1 when it
     # never expires, nil when it is missing or expired.
     def ttl(coll, id)
-      json(request("GET", "#{id_path(coll, id)}/ttl", nil)).fetch("ttl")
+      json(request("GET", doc_path(coll, "_ttl", id), nil)).fetch("ttl")
     rescue NotFound
       nil
     end
 
     # Streams the changes of a collection for a named consumer: first the ones
     # after its last ack, then the new ones. Yields one Hash per event (pos,
-    # op, id, v, doc). where is the same Hash/Range conditions as find, and
-    # only the matching events come; limit ends the stream after that many
-    # events. Without a block it returns an Enumerator. Delivery is at least
-    # once: ack the pos of an event when its work is done.
+    # op, _id, _v, doc). query is the same conditions as a search, and only
+    # the matching events come; limit ends the stream after that many events.
+    # Without a block it returns an Enumerator. Delivery is at least once: ack
+    # the pos of an event when its work is done.
     #
     # The stream raises WatchLagged when the consumer is too slow, and
     # HistoryTruncated when its position is older than the kept WAL; call
     # watch again for the first, start over for the second.
-    def watch(coll, consumer, where: nil, from: nil, limit: nil, &block)
-      return enum_for(:watch, coll, consumer, where: where, from: from, limit: limit) unless block
+    def watch(coll, consumer, query: nil, from: nil, limit: nil, &block)
+      return enum_for(:watch, coll, consumer, query: query, from: from, limit: limit) unless block
 
       body = { consumer: consumer, from: from, limit: limit }.compact
-      body[:where] = Conditions.equality(where) if where
+      body[:query] = Conditions.equality(query) if query
 
-      with_ready_retry { stream_events("#{path(coll)}/watch", body, &block) }
+      with_ready_retry { stream_events("#{path(coll)}/_watch", body, &block) }
     end
 
     # Stores the position of an event for its consumer: the next watch with
     # that consumer name starts after it.
     def ack(coll, consumer, pos)
-      json(post(coll, "ack", consumer: consumer, pos: pos))
+      json(post(coll, "_ack", consumer: consumer, pos: pos))
     end
 
     # Removes every document of the collection and keeps its definition and
     # indexes. The cost does not depend on the number of documents. Watch
     # consumers get one event with op "truncate". {"collection", "truncated"}
     def truncate(coll)
-      json(request("POST", "#{path(coll)}/truncate?confirm=#{confirm(coll)}", nil))
+      json(request("POST", "#{path(coll)}/_truncate?confirm=#{confirm(coll)}", nil))
     end
 
     # Removes the collection: its documents, its definition and the positions
@@ -159,24 +193,64 @@ module Ndb
       json(request("DELETE", "#{path(coll)}?confirm=#{confirm(coll)}", nil))
     end
 
+    # PUT /{coll}: creates the collection or merges the definition into it.
     def ensure_collection(coll, definition = {})
       json(request("PUT", path(coll), definition))
     end
 
+    # PUT /{coll}/_indexes/{field}: adds an index, built in the background.
+    def ensure_index(coll, field)
+      json(request("PUT", doc_path(coll, "_indexes", field), nil)).fetch("indexes")
+    end
+
+    # DELETE /{coll}/_indexes/{field}.
+    def drop_index(coll, field)
+      json(request("DELETE", doc_path(coll, "_indexes", field), nil)).fetch("indexes")
+    end
+
+    # GET /{coll}/_stats: rows, indexes, files and bytes of one collection.
+    def collection_stats(coll)
+      json(request("GET", "#{path(coll)}/_stats", nil))
+    end
+
+    # POST /{coll}/_compact: seals and merges every segment now.
+    def compact(coll)
+      json(post(coll, "_compact", nil))
+    end
+
+    # GET /_health: true when the daemon is ready; raises NotReady while it
+    # recovers. It does not wait.
+    def health
+      request("GET", "/_health", nil, wait: false)
+
+      true
+    end
+
+    # GET /_collections: one definition per collection.
     def collections
-      json(request("GET", "/v1/c", nil)).fetch("collections")
+      json(request("GET", "/_collections", nil)).fetch("collections")
+    end
+
+    # GET /_stats: every collection, totals, version and uptime.
+    def stats
+      json(request("GET", "/_stats", nil))
+    end
+
+    # GET /_size: disk use per collection and per part.
+    def size
+      json(request("GET", "/_size", nil))
     end
 
     # Uploads an IO (streamed) and returns the reference hash.
     def put_blob(io, content_type = nil)
-      json(request("PUT", "/v1/blobs", io, headers: { "Content-Type" => content_type || "application/octet-stream" }))
+      json(request("POST", "/_blobs", io, headers: { "Content-Type" => content_type || "application/octet-stream" }))
     end
 
     # The bytes of a blob, or of its range from..to (inclusive).
     def get_blob(ref, from: nil, to: nil)
       headers = from ? { "Range" => "bytes=#{from}-#{to}" } : {}
 
-      request("GET", "/v1/blobs/#{ref}", nil, headers: headers).body
+      request("GET", "/_blobs/#{ref}", nil, headers: headers).body
     end
 
     private
@@ -185,51 +259,50 @@ module Ndb
       value.to_s.strip.empty? ? nil : value
     end
 
-    def write_query(many: false, upsert: false, expire_in: nil)
-      params = {}
-
-      params["many"] = "true" if many
-      params["upsert"] = "true" if upsert
-      params["expire_in"] = expire_in.to_s if expire_in
-
-      params.empty? ? "" : "?#{URI.encode_www_form(params)}"
+    # The body of a write, without the options that are off.
+    def write_body(**parts)
+      parts.reject { |_, v| v.nil? || v == false }
     end
 
-    # The filter and the operators in one body. An operator key without "$"
-    # would be read as a filter key by the daemon, so it is refused here.
-    def merge_update(filter, ops)
+    # An operator key without "$" is a mistake, so it is refused here.
+    def check_ops(ops)
       ops.each_key do |key|
         next if key.to_s.start_with?("$")
 
         raise ArgumentError, "update operator #{key.to_s.inspect} must start with $ (for example \"$set\")"
       end
 
-      filter.to_h.transform_keys(&:to_s).merge(ops.to_h.transform_keys(&:to_s))
+      ops
     end
 
-    def post(coll, verb, body)
-      request("POST", "#{path(coll)}/#{verb}", body)
+    def post(coll, action, body)
+      request("POST", "#{path(coll)}/#{action}", body)
     end
 
     def path(coll)
-      "/v1/c/#{URI.encode_www_form_component(coll).gsub('+', '%20')}"
+      "/#{escape(coll)}"
+    end
+
+    def escape(segment)
+      URI.encode_www_form_component(segment).gsub("+", "%20")
     end
 
     def confirm(coll)
       URI.encode_www_form_component(coll)
     end
 
-    def id_path(coll, id)
-      "#{path(coll)}/#{URI.encode_www_form_component(id).gsub('+', '%20')}"
+    def doc_path(coll, action, id)
+      "#{path(coll)}/#{action}/#{escape(id)}"
     end
 
-    def request(method, target, body, headers: {})
+    def request(method, target, body, headers: {}, wait: true)
       payload = encode(body)
       headers = authorize(headers)
 
       safe = idempotent?(method, target) && !payload.respond_to?(:read)
+      timeout = wait ? @ready_timeout : 0
 
-      with_ready_retry(idempotent: safe) do
+      with_ready_retry(idempotent: safe, timeout: timeout) do
         response = @pool.with { |conn| conn.request(method, target, payload, headers: headers, idempotent: safe) }
 
         raise_api_error(response) if response.status >= 300
@@ -239,20 +312,20 @@ module Ndb
     end
 
     def idempotent?(method, target)
-      method == "GET" || target.match?(%r{/(find|count|aggregate|get|stats)\z})
+      method == "GET" || target.match?(%r{/(_search|_count|_aggregate)\z})
     end
 
     # Runs the block again while the daemon is not ready. It repeats only
     # when the request certainly was not processed (not_ready, a failed
     # connect), or for a read when the connection dropped. Never a timeout.
-    def with_ready_retry(idempotent: false)
-      deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + @ready_timeout
+    def with_ready_retry(idempotent: false, timeout: @ready_timeout)
+      deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + timeout
       delay = RETRY_BASE
 
       begin
         yield
       rescue NotReady, ConnectError, ConnectionError => e
-        raise if @ready_timeout <= 0 || e.is_a?(TimeoutError)
+        raise if timeout <= 0 || e.is_a?(TimeoutError)
         raise unless e.is_a?(NotReady) || e.is_a?(ConnectError) || idempotent
 
         wait = (delay / 2) + (rand * delay / 2)

data/lib/ndb/collection.rb

@@ -0,0 +1,269 @@
+# frozen_string_literal: true
+
+module Ndb
+  # One collection of the daemon, with the query builder. The first call
+  # declares it (PUT /{coll}) with its definition. A name with a "*" at the
+  # end is a collection pattern ("logs-2026-*"): it reads every collection
+  # whose name starts with the prefix, declares nothing, and its writes raise
+  # Ndb::Error with code pattern_read_only.
+  #
+  #   db = Ndb.connect("unix:///run/ndb/ndb.sock")
+  #   calls = db.collection("calls", time: "ts", indexes: ["status"])
+  #   calls.insert(status: "answered", ts: Time.now)
+  #   calls.query(status: "answered").sort(ts: -1).limit(10).all
+  class Collection
+    attr_reader :client, :name, :definition
+
+    def initialize(client, name, definition = {})
+      @client = client
+      @name = name
+      @definition = definition
+      @ensured = false
+    end
+
+    def pattern? = name.include?("*")
+
+    # A Hash is one document (_doc) and returns {"_id", "_v"}; an Array is
+    # many in one transaction (_bulk) and returns the results. expire_in
+    # applies to every document.
+    def insert(doc_or_docs = nil, expire_in: nil, **fields)
+      doc_or_docs ||= fields
+      ready!
+
+      return client.bulk(name, doc_or_docs, expire_in: expire_in) if doc_or_docs.is_a?(Array)
+      return client.bulk(name, [doc_or_docs], expire_in: expire_in).first if expire_in
+
+      client.insert(name, doc_or_docs)
+    end
+
+    # Writes a document by its _id: creates it when the _id is free, and
+    # merges its fields into the stored document when it exists (the other
+    # fields stay), like doc_as_upsert in Elasticsearch. No _id is an insert.
+    # Returns {"_id", "_v"}.
+    def upsert(doc = nil, expire_in: nil, **fields)
+      upsert_all([doc || fields], expire_in: expire_in).first
+    end
+
+    # upsert for many documents in one transaction, all or nothing.
+    def upsert_all(docs, expire_in: nil)
+      ready!
+      client.bulk(name, docs, expire_in: expire_in, upsert: true)
+    end
+
+    # The document with this _id, or nil.
+    def get(id)
+      ready!
+      client.get(name, id)
+    end
+
+    # Applies the write operators to the document with this _id (atomic).
+    # guard adds conditions on the same document, such as { _v: 7 }.
+    def update(id, ops, guard: nil, upsert: false, expire_in: nil)
+      ready!
+      client.update(name, id, ops, guard: guard, upsert: upsert, expire_in: expire_in)
+    end
+
+    # The atomic update by query: the first match, or every match with many.
+    def update_by(query, ops, many: false, upsert: false, expire_in: nil)
+      ready!
+      client.update_by(name, query, ops, many: many, upsert: upsert, expire_in: expire_in)
+    end
+
+    def delete(id)
+      ready!
+      client.delete(name, id)
+    end
+
+    def delete_by(query = nil, many: false, **conditions)
+      query ||= conditions
+      ready!
+      client.delete_by(name, query, many: many)
+    end
+
+    # A search; the conditions use the Ruby forms of Ndb::Conditions
+    # (Ranges, Times, Regexps, an Array as $in).
+    def query(conditions = nil, **keywords)
+      Query.new(self).query(conditions || keywords)
+    end
+
+    # A query built elsewhere, such as from a search form, sent as is.
+    def raw_query(hash)
+      Query.new(self).raw_query(hash)
+    end
+
+    # How many documents match.
+    def count(conditions = nil, **keywords)
+      query(conditions || keywords).count
+    end
+
+    # A whole search body built elsewhere: { "query", "sort", "limit",
+    # "after", "count" }. Returns an Ndb::Page.
+    def search(body)
+      ready!
+      client.search(name, body)
+    end
+
+    def aggregate(pipeline, **opts)
+      ready!
+      client.aggregate(name, { pipeline: pipeline }.merge(opts))
+    end
+
+    # Each event Hash also has "consumer", so ack(event) needs nothing else.
+    def watch(consumer, query: nil, from: nil, limit: nil, &block)
+      return enum_for(:watch, consumer, query: query, from: from, limit: limit) unless block
+
+      ready!
+      client.watch(name, consumer, query: query, from: from, limit: limit) do |event|
+        block.call(event.merge("consumer" => consumer))
+      end
+    end
+
+    # Stores the position of an event from watch for its consumer: the next
+    # watch with that consumer name starts after it.
+    def ack(event, consumer = nil)
+      client.ack(name, consumer || event.fetch("consumer"), event.fetch("pos"))
+    end
+
+    def expire(id, **time)
+      ready!
+      client.expire(name, id, **time)
+    end
+
+    def persist(id)
+      ready!
+      client.persist(name, id)
+    end
+
+    def ttl(id)
+      ready!
+      client.ttl(name, id)
+    end
+
+    def truncate = client.truncate(name)
+
+    def drop
+      @ensured = false
+      client.drop(name)
+    end
+
+    def stats
+      ready!
+      client.collection_stats(name)
+    end
+
+    def compact
+      ready!
+      client.compact(name)
+    end
+
+    def ensure_index(field)
+      ready!
+      client.ensure_index(name, field)
+    end
+
+    def drop_index(field)
+      ready!
+      client.drop_index(name, field)
+    end
+
+    # Declares the collection once (not a pattern).
+    def ready!
+      return if @ensured || pattern?
+
+      client.ensure_collection(name, definition)
+      @ensured = true
+    end
+  end
+
+  # An immutable search on a collection: every step returns a new Query.
+  class Query
+    include Enumerable
+
+    def initialize(collection, clauses: [], options: {})
+      @collection = collection
+      @clauses = clauses
+      @options = options
+    end
+
+    # Adds conditions (Ruby forms), ANDed with the ones already there.
+    def query(conditions)
+      return self if conditions.empty?
+
+      with(clauses: @clauses + [Conditions.equality(conditions)])
+    end
+
+    # Adds a query built elsewhere, sent as is.
+    def raw_query(hash)
+      with(clauses: @clauses + [Conditions.json_value(hash)])
+    end
+
+    def sort(spec) = with(sort: spec.to_h { |k, v| [k.to_s, direction(v)] })
+    def limit(count) = with(limit: count)
+    def skip(count) = with(skip: count)
+    def after(token) = with(after: token)
+    def fields(*names) = with(fields: names.flatten.map(&:to_s))
+
+    # The rows of one request: one page with limit, every row without it.
+    def all = page.rows
+
+    def first = limit(1).all.first
+
+    # One page: rows, next cursor, total (with count: true via #page(count: true)).
+    def page(count: false)
+      body = body_hash
+      body["count"] = true if count
+
+      @collection.search(body)
+    end
+
+    # How many documents match (POST /{coll}/_count).
+    def count
+      @collection.ready!
+      @collection.client.count(@collection.name, where)
+    end
+
+    # Every row, following the cursor page by page (limit is the page size).
+    def each(&block)
+      return enum_for(:each) unless block
+
+      query = self
+
+      loop do
+        result = query.page
+        result.rows.each(&block)
+
+        break unless result.next
+
+        query = query.after(result.next).with(skip: nil)
+      end
+    end
+
+    def body_hash
+      body = { "query" => where }.merge(@options.compact.transform_keys(&:to_s))
+      body.delete("query") if body["query"].empty?
+      body
+    end
+
+    protected
+
+    def with(**options)
+      clauses = options.delete(:clauses) || @clauses
+
+      self.class.new(@collection, clauses: clauses, options: @options.merge(options))
+    end
+
+    private
+
+    def where = Conditions.combine(@clauses)
+
+    def direction(value)
+      %w[desc -1].include?(value.to_s.downcase) ? -1 : 1
+    end
+  end
+
+  # A client for addr: unix:///path/to/ndb.sock or tcp://host:port. Without
+  # addr it reads NDB_URL. See Ndb::Client for the options.
+  def self.connect(addr = nil, **opts)
+    Client.new(url: addr, **opts)
+  end
+end

data/lib/ndb/connection.rb

@@ -4,7 +4,7 @@ require "socket"
 require "uri"
 
 module Ndb
-  Response = Struct.new(:status, :headers, :body)
+  Response = Data.define(:status, :headers, :body)
 
   # Where the daemon listens: a unix socket path, or a TCP host and port.
   # The host name is resolved each time a connection opens, so a changed
@@ -138,7 +138,7 @@ module Ndb
     def send_request(method, target, body, headers)
       open_socket unless @socket
 
-      head = +"#{method} #{target} HTTP/1.1\r\nHost: #{@endpoint.host_header}\r\nAccept: application/json\r\n"
+      head = +"#{method} #{target} HTTP/1.1\r\nHost: #{@endpoint.host_header}\r\nAccept: application/x-ndjson\r\n"
 
       { "Content-Type" => "application/json" }.merge(headers).each { |name, value| head << "#{name}: #{value}\r\n" }
 

data/lib/ndb/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Ndb
-  VERSION = "0.4.0"
+  VERSION = "1.0.0"
 end

data/lib/ndb.rb

@@ -6,3 +6,4 @@ require "ndb/errors"
 require "ndb/conditions"
 require "ndb/connection"
 require "ndb/client"
+require "ndb/collection"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ndb
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 1.0.0
 platform: ruby
 authors:
 - thadeu
@@ -20,6 +20,7 @@ files:
 - README.md
 - lib/ndb.rb
 - lib/ndb/client.rb
+- lib/ndb/collection.rb
 - lib/ndb/conditions.rb
 - lib/ndb/connection.rb
 - lib/ndb/errors.rb