ndb-rails 0.1.0 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 407a37e97a7c0f50f39d4ca365e59bd9ecec788a9fe357e4ff4fbc61add79962
-  data.tar.gz: 0de5474707d6bf68b55d1a44f56b612a8d5a59bc24cd7e96ff8066af00e98f8b
+  metadata.gz: cb272d6aadee253c818c30de3c3820c510903f06d42dac23dd56837bb585280a
+  data.tar.gz: 4ccfbb876e868ebea9b229a5ef44292ad3240ed55933a46e3a5a432158f25012
 SHA512:
-  metadata.gz: 5eeed45c9f972b596dd588122600f1f7729ca07957ca771a8af0d0ca34085aa8b3c8e76d898a9605e034a6402c8370bdf57e209b2da5320a84d5904176fbeb50
-  data.tar.gz: d52fa0af1af45659950dd152a68a5561aa311b98c7b3296f70df1185d6071b70740eae7f95763bb58d1aafc35984a6aa11cf04b822d05ab98ed2a02de208a88b
+  metadata.gz: 0c24be778d6fc65e219b829d7c8219e810ecc3505a39c0670077e07a17c60a37824d5814a6869dd6926fa52eed09a85b6615622779be126e8c87e582ece20a0d
+  data.tar.gz: 6360ba54f2208b4b438f269cfbbfd172fe0ebf9fd6907df523369d5241f88eec5a36ca617504dbde0553624fff00156b09c7cb892b05790d34e00cc4dde51f55

data/README.md

@@ -112,13 +112,21 @@ call = Call.find("c1")        # raises Ndb::NotFound
 Call.find_by(status: "failed") # nil when missing
 Call.find_by!(status: "x")     # raises NotFound
 
-call.update!(status: "done")  # sends only the changed fields
+call.update!(status: "done")  # $set of the changed fields only, filtered by id and version
 call.version                   # the document version
 
+call.increment!(:duration)     # atomic $inc on the daemon, refreshes the attribute and the version
+call.decrement!(:duration, 5)
+
 call.destroy
+
+# Atomic writes on every match of a where (it needs the id or an indexed field).
+# update takes the operators of the client and returns the number of modified documents.
+Call.where(status: "failed").update({ "$set" => { "status" => "retry" } })
+Call.where(status: "retry").delete       # returns how many it deleted
 ```
 
-- Every load asks for the version (`_v`). `save`, `update` and `destroy` send it back as `if_version`. If another writer changed the document first, you get `Ndb::StaleObjectError` (code `version_conflict`).
+- Every load asks for the version (`_v`). `save`, `update` and `destroy` put the id and `_v` in the filter of the write. If another writer changed the document first, the filter matches nothing and you get `Ndb::StaleObjectError` (code `stale_object`).
 - Setting an attribute to `nil` and saving removes the field from the document.
 - A duplicate id raises `RecordNotUnique` (code `duplicate_id`).
 - `persisted?`, `new_record?`, `destroyed?`, `id` and `version` work as in ActiveRecord.
@@ -132,6 +140,32 @@ rescue Ndb::StaleObjectError
 end
 ```
 
+## Expiry (TTL like Redis)
+
+Give the model an `expires: true` attribute. The daemon sets it for you.
+
+```ruby
+class Session < Ndb::Record
+  primary_key :sid
+
+  attribute :user, :string
+  attribute :expires_at, :datetime, expires: true
+end
+
+Session.create!({ sid: "s1", user: "u" }, expire_in: 1.hour)   # also create, save, update (seconds, a Duration or "30m")
+session = Session.find("s1")
+
+session.expire(30.minutes)   # EXPIRE: sets expires_at in the object, other unsaved changes stay
+session.ttl                  # => 1799, -1 (never expires), or nil (missing or expired)
+session.persist_ttl!         # PERSIST: removes the expiry
+```
+
+- `persist_ttl!` is not called `persist!`: ActiveModel already uses `persisted?` for something else.
+- `expire_in:` on `create`, `save` and `update` goes in the same write. `expire_in` wins over the `expires_at` value. After a write with `expire_in`, load the record again (`Session.find`) to see `expires_at`.
+- `expire` and `persist_ttl!` raise `Ndb::NotFound` for a missing or expired document, and `Ndb::TtlNotEnabled` when the model has no `expires: true` attribute.
+- The class-level `ttl 24.hours` stays: every document expires that long after its time field. `expire` and `persist_ttl!` raise `Ndb::TtlDerived` for it.
+- Disk space of expired documents returns after compaction, and fully only with `wal_retention`; see [Freeing disk](https://github.com/thadeu/ndb/blob/main/docs/dev/configuration.md#freeing-disk).
+
 ## Query
 
 Relations are lazy and immutable. The request goes out when you call `to_a`, `first`, `each`, `count` and so on.
@@ -161,7 +195,7 @@ Call.where(status: "answered").first(5)
 Call.where(status: "failed").exists?
 ```
 
-- Chained `where` calls are ANDed. `$regex` does not exist in the daemon yet. See the [query language](https://github.com/thadeu/ndb/blob/main/docs/dev/query.md) for the full operator list.
+- Chained `where` calls are ANDed. See the [query language](https://github.com/thadeu/ndb/blob/main/docs/dev/query.md) for the full operator list.
 - The order of the `order` keys is the order of the sort. The id is always the last key.
 - `page(n).per(size)` sets offset paging. The default page size is 25.
 - `pluck` asks the daemon for only those fields.
@@ -295,7 +329,7 @@ A blob behaves like a read-only IO (`read`, `each_chunk`, `eof?`, `rewind`). A l
 
 ## Errors
 
-The transport and daemon errors are defined by the gem `ndb`. They all live in the `Ndb` module (`Ndb::Error`, `Ndb::NotFound`, ...). For an ActiveRecord-like `rescue`, this gem adds two aliases: `Ndb::StaleObjectError` is `Ndb::VersionConflict`, and `Ndb::RecordNotUnique` is `Ndb::UniqueViolation`. They are the same classes, so a `rescue` of either name catches the same error. `Ndb::RecordInvalid` and `Ndb::RecordNotSaved` exist only here, and they inherit from `Ndb::Error`. Every error has `code` (the daemon code) and `status` (the HTTP status).
+The transport and daemon errors are defined by the gem `ndb`. They all live in the `Ndb` module (`Ndb::Error`, `Ndb::NotFound`, ...). For an ActiveRecord-like `rescue`, this gem adds one alias: `Ndb::RecordNotUnique` is `Ndb::UniqueViolation` (the same class). `Ndb::StaleObjectError`, `Ndb::RecordInvalid` and `Ndb::RecordNotSaved` exist only here, and they inherit from `Ndb::Error`. Every error has `code` (the daemon code) and `status` (the HTTP status).
 
 | Class | Code |
 |---|---|
@@ -303,7 +337,8 @@ The transport and daemon errors are defined by the gem `ndb`. They all live in t
 | `NotReady` | `not_ready` |
 | `NotFound` | `not_found` |
 | `UnknownCollection` | `unknown_collection` |
-| `StaleObjectError` | `version_conflict` |
+| `StaleObjectError` | `stale_object` (raised by the gem, not sent by the daemon) |
+| `InvalidUpdate` and its subclasses | `unknown_operator`, `no_update`, `conflicting_update`, `invalid_field`, `unindexed_many`, `empty_filter` |
 | `RecordNotUnique` | `unique_violation`, `duplicate_id` |
 | `CatalogConflict` | `catalog_conflict` |
 | `CatalogInvalid` | `catalog_invalid` |
@@ -312,6 +347,7 @@ The transport and daemon errors are defined by the gem `ndb`. They all live in t
 | `InvalidCursor` | `cursor_invalid`, `cursor_mismatch`, `cursor_expired` |
 | `WatchLagged` | `watch_lagged` |
 | `HistoryTruncated` | `history_truncated` |
+| `TtlNotEnabled`, `TtlDerived` | `ttl_not_enabled`, `ttl_derived` |
 | `RecordInvalid` | ActiveModel validation failed (has `record`) |
 | `RecordNotSaved` | `save!` returned false without errors |
 | `ConnectionError` | cannot reach the daemon. Subclasses: `ConnectError` (nothing was sent), `TimeoutError` |

data/lib/ndb/rails/errors.rb

@@ -1,11 +1,14 @@
 # frozen_string_literal: true
 
 module Ndb
-  # ActiveRecord-like names for two daemon errors. They are the same classes,
-  # so a rescue of either name catches the same error.
-  StaleObjectError = VersionConflict
+  # ActiveRecord-like name for a daemon error. It is the same class, so a
+  # rescue of either name catches the same error.
   RecordNotUnique = UniqueViolation
 
+  # A save or destroy matched nothing: the record changed or was deleted
+  # since it was loaded (its "_v" guard did not match).
+  class StaleObjectError < Error; end
+
   # Raised by save!/create!/update! when validations fail.
   class RecordInvalid < Error
     attr_reader :record

data/lib/ndb/rails/persistence.rb

@@ -3,7 +3,8 @@
 module Ndb
   class RecordNotSaved < Error; end
 
-  # Writes: dirty attributes -> set/unset, the loaded version -> if_version.
+  # Writes: dirty attributes -> $set/$unset, filtered by the id and the loaded
+  # version ("_v"). A write that matches nothing is a stale record.
   module Persistence
     extend ActiveSupport::Concern
 
@@ -12,24 +13,28 @@ module Ndb
     end
 
     class_methods do
-      def create(attrs = {}, &block)
-        return attrs.map { |a| create(a, &block) } if attrs.is_a?(Array)
+      def create(attrs = {}, expire_in: nil, **rest, &block)
+        return attrs.map { |a| create(a, expire_in: expire_in, &block) } if attrs.is_a?(Array)
+
+        attrs = attrs.merge(rest)
 
         record = new(attrs)
 
         yield record if block
-        record.save
+        record.save(expire_in: expire_in)
 
         record
       end
 
-      def create!(attrs = {}, &block)
-        return attrs.map { |a| create!(a, &block) } if attrs.is_a?(Array)
+      def create!(attrs = {}, expire_in: nil, **rest, &block)
+        return attrs.map { |a| create!(a, expire_in: expire_in, &block) } if attrs.is_a?(Array)
+
+        attrs = attrs.merge(rest)
 
         record = new(attrs)
 
         yield record if block
-        record.save!
+        record.save!(expire_in: expire_in)
 
         record
       end
@@ -71,43 +76,134 @@ module Ndb
       persisted? ? [id] : nil
     end
 
-    def save(validate: true)
+    # expire_in (seconds, an ActiveSupport::Duration or "30m") makes the daemon
+    # set the expires field to now + expire_in in the same write. The model
+    # needs an `expires: true` attribute (Ndb::TtlNotEnabled otherwise).
+    def save(validate: true, expire_in: nil)
       return false if validate && !valid?(new_record? ? :create : :update)
 
+      @expire_in = wire_duration(expire_in)
+
       run_callbacks(:save) { new_record? ? create_record : update_record } ? true : false
+    ensure
+      @expire_in = nil
     end
 
     def save!(**)
       save(**) || raise(errors.any? ? RecordInvalid.new(self) : RecordNotSaved.new("Failed to save the record"))
     end
 
-    def update(attrs)
-      assign_attributes(attrs)
+    def update(attrs = {}, expire_in: nil, **rest)
+      assign_attributes(attrs.merge(rest))
+
+      save(expire_in: expire_in)
+    end
+
+    def update!(attrs = {}, expire_in: nil, **rest)
+      assign_attributes(attrs.merge(rest))
 
-      save
+      save!(expire_in: expire_in)
     end
 
-    def update!(attrs)
-      assign_attributes(attrs)
+    # Makes the stored document expire in a duration (like EXPIRE in Redis):
+    # `session.expire(30.minutes)`. The version and the expires attribute are
+    # refreshed; other unsaved changes stay unsaved. Raises Ndb::NotFound for
+    # a missing or expired document and Ndb::TtlNotEnabled without an
+    # `expires: true` attribute.
+    def expire(duration)
+      result = Rails.client.expire(self.class.collection_name, id, in: wire_duration(duration))
 
-      save!
+      refresh_expiry(result, Time.iso8601(result["expires_at"]))
+    end
+
+    # Removes the expiry of the stored document (PERSIST in Redis). It is not
+    # called `persist!` or `persisted?`, which mean something else in
+    # ActiveModel.
+    def persist_ttl!
+      refresh_expiry(Rails.client.persist(self.class.collection_name, id), nil)
+    end
+
+    # The seconds left (Integer), -1 when the document never expires, nil when
+    # it is missing or expired.
+    def ttl
+      Rails.client.ttl(self.class.collection_name, id)
     end
 
     def destroy
       result = run_callbacks(:destroy) do
-        self.class.ndb_delete(id, if_version: version) if persisted?
+        delete_record if persisted?
         @destroyed = true
       end
 
       result ? self : false
     end
 
+    # Adds by (default 1) to a number attribute in one atomic write on the
+    # daemon, and refreshes the attribute and the version. Other unsaved
+    # changes stay unsaved.
+    def increment!(attribute, by = 1)
+      add_to!(attribute, by)
+    end
+
+    def decrement!(attribute, by = 1)
+      add_to!(attribute, -by)
+    end
+
     def ==(other)
       other.instance_of?(self.class) && !id.nil? && id == other.id
     end
 
     private
 
+    def wire_duration(value)
+      value.is_a?(ActiveSupport::Duration) ? value.to_i : value
+    end
+
+    def refresh_expiry(result, expires_at)
+      @version = result["v"]
+
+      field = self.class.expires_field
+
+      return self unless field
+
+      public_send("#{field}=", expires_at)
+      clear_attribute_changes([field])
+
+      self
+    end
+
+    def record_filter
+      filter = { self.class.primary_key => id }
+
+      filter["_v"] = version if version
+
+      filter
+    end
+
+    def delete_record
+      deleted = self.class.ndb_delete(record_filter)["deleted"]
+
+      raise StaleObjectError.new("#{self.class.name} #{id} changed or was deleted", code: "stale_object") if deleted.zero?
+    end
+
+    def add_to!(attribute, by)
+      name = attribute.to_s
+
+      raise RecordNotSaved, "Cannot update a record that is not saved" unless persisted?
+      raise ArgumentError, "#{name} is not an attribute of #{self.class.name}" unless self.class.attribute_names.include?(name)
+
+      result = self.class.ndb_update({ self.class.primary_key => id }, { "$inc" => { name => by } })
+
+      raise NotFound.new("#{self.class.name} #{id} was deleted", code: "not_found") if result["matched"].zero?
+
+      @version = result["doc"]["_v"]
+
+      public_send("#{name}=", result["doc"][name])
+      clear_attribute_changes([name])
+
+      self
+    end
+
     def loaded!(version)
       @persisted = true
       @version = version
@@ -117,7 +213,7 @@ module Ndb
 
     def create_record
       run_callbacks(:create) do
-        result = self.class.ndb_insert(insert_document)
+        result = self.class.ndb_insert(insert_document, expire_in: @expire_in)
 
         self.id = result["id"] if id.nil?
         loaded!(result["v"])
@@ -130,10 +226,12 @@ module Ndb
       run_callbacks(:update) do
         set, unset = changed_fields
 
-        unless set.empty? && unset.empty?
-          result = self.class.ndb_update(id, set: set, unset: unset, if_version: version)
+        unless set.empty? && unset.empty? && @expire_in.nil?
+          result = self.class.ndb_update(record_filter, update_ops(set, unset), expire_in: @expire_in)
 
-          @version = result["v"]
+          raise StaleObjectError.new("#{self.class.name} #{id} changed or was deleted", code: "stale_object") if result["matched"].zero?
+
+          @version = result["doc"]["_v"]
         end
 
         changes_applied
@@ -142,6 +240,15 @@ module Ndb
       end
     end
 
+    def update_ops(set, unset)
+      ops = {}
+
+      ops["$set"] = set unless set.empty?
+      ops["$unset"] = unset.to_h { |name| [name, true] } unless unset.empty?
+
+      ops.empty? ? { "$set" => {} } : ops
+    end
+
     def insert_document
       attribute_names.each_with_object({}) do |name, doc|
         value = json_attribute(name)

data/lib/ndb/rails/record.rb

@@ -124,7 +124,7 @@ module Ndb
         end
       end
 
-      # Every load asks for "_v", so a loaded record can send if_version.
+      # Every load asks for "_v", so a loaded record can guard its write with it.
       def ndb_find(body)
         ensure_collection!
 
@@ -137,22 +137,23 @@ module Ndb
         Rails.client.aggregate(collection_name, body)
       end
 
-      def ndb_insert(doc)
+      def ndb_insert(doc, expire_in: nil)
         ensure_collection!
 
-        Rails.client.insert(collection_name, [doc]).first
+        Rails.client.insert(collection_name, [doc], expire_in: expire_in).first
       end
 
-      def ndb_update(id, **change)
+      # Atomic update by filter, see Ndb::Client#update.
+      def ndb_update(filter, ops, **options)
         ensure_collection!
 
-        Rails.client.update(collection_name, id, **change)
+        Rails.client.update(collection_name, filter, ops, **options)
       end
 
-      def ndb_delete(id, if_version: nil)
+      def ndb_delete(filter, **options)
         ensure_collection!
 
-        Rails.client.delete(collection_name, id, if_version: if_version)
+        Rails.client.delete(collection_name, filter, **options)
       end
 
       private

data/lib/ndb/rails/relation.rb

@@ -78,6 +78,20 @@ module Ndb
       query
     end
 
+    # Atomic writes. The daemon matches the where and writes in one step on
+    # every match, so the where needs the id or an indexed field
+    # (Ndb::UnindexedMany otherwise). ops is the operator Hash of
+    # Ndb::Client#update ({ "$inc" => { "n" => 1 } }); returns the number of
+    # modified documents.
+    def update(ops, expire_in: nil)
+      klass.ndb_update(Conditions.combine(values[:where]), ops, many: true, expire_in: expire_in).fetch("modified")
+    end
+
+    # Deletes every match (same rule as update) and returns how many.
+    def delete
+      klass.ndb_delete(Conditions.combine(values[:where]), many: true).fetch("deleted")
+    end
+
     # Loaders
 
     def load

data/lib/ndb/rails/version.rb

@@ -2,6 +2,6 @@
 
 module Ndb
   module Rails
-    VERSION = "0.1.0"
+    VERSION = "0.1.2"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ndb-rails
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.2
 platform: ruby
 authors:
 - thadeu