elasticsearch_record 1.5.2 → 1.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9a53ead4edbffa7bb74c8021008343f80813512f89b6580717d7d4335ce238ba
-  data.tar.gz: 26603c0b2de3b938c4239401652d4986088be1d29b1c039c2feb0c48b83a5207
+  metadata.gz: b29c6db7894f8365eb5a5922633a4adf53a793006c3ffa2fb381dd9ae171b124
+  data.tar.gz: 72c7f4260b76be5743e061838df5051006c82a883172d29f2e0442b6215ac4ed
 SHA512:
-  metadata.gz: 8d60b93db8b944ac6b0b42c91e411d6ae9673f773feeab7fbc9097e519663401a349936a271f6949d65f46485198cce44c349c38d3d2f668e2f3ee6148417264
-  data.tar.gz: 9e3a2700baa40c8bc4b6a72c6a94895f3beb030294955ca6d1a168349c81b289d80bf2d93cfa5411f0a8c2a74dd3b183ece5ee64ef3deef7ff2f0f440aef32dc
+  metadata.gz: 7e09f30a077c81524c800a5694947d947da7a0a9273a4304592221d2c191c8a8582fbd94e65990ddf21cdbb23d74d5e7d4e47e53e14139c362fda91468aae182
+  data.tar.gz: bb03f9c4fa2749ccd23be70de65a1480b2d0a7cfa89ef51f9a3c3c2cd7e0810708e9e1e8050d7de033bfc670d5a127c75349f1f1380578246107395746e41652

data/README.md

@@ -53,6 +53,22 @@ Or install it yourself as:
   * logs Elasticsearch API-calls
   * shows Runtime in logs
 
+## Notice
+Since ActiveRecord does not have any configuration option to support transactions and 
+Elasticsearch does **NOT** support transactions, it may be risky to ignore them.
+
+As a default, transactions are 'silently swallowed' to not break any existing applications...
+
+To raise an exception while using transactions on a ElasticsearchRecord model, the following flag can be enabled.
+However enabling this flag will surely fail transactional tests _(prevent this with 'use_transactional_tests=false')_
+
+```ruby
+# config/initializers/elasticsearch_record.yml
+
+# enable transactional exceptions
+ElasticsearchRecord.error_on_transaction = true
+```
+
 ## Setup
 
 ### a) Update your **database.yml** and add a elasticsearch connection:
@@ -224,6 +240,7 @@ total = scope.total
 - configure
 - aggregate
 - refresh
+- timeout
 - query
 - filter
 - must_not
@@ -260,6 +277,7 @@ _see simple documentation about these methods @ [rubydoc](https://rubydoc.info/g
 - composite
 - point_in_time
 - pit_results
+- pit_delete
 
 _see simple documentation about these methods @ [rubydoc](https://rubydoc.info/gems/elasticsearch_record/ElasticsearchRecord/Relation/ResultMethods)_
 
@@ -366,13 +384,26 @@ SearchUser.api.mappings
 SearchUser.api.insert([{name: 'Hans', age: 34}, {name: 'Peter', age: 22}])
 ```
 
+### dangerous methods
 * open!
 * close!
 * refresh!
 * block!
 * unblock!
+
+### dangerous methods with args
+* create!(...)
+* clone!(...)
+* rename!(...)
+* backup!(...)
+* restore!(...)
+* reindex!(...)
+
+### dangerous methods with confirm parameter
 * drop!(confirm: true)
 * truncate!(confirm: true)
+
+### table methods
 * mappings
 * metas
 * settings
@@ -380,17 +411,19 @@ SearchUser.api.insert([{name: 'Hans', age: 34}, {name: 'Peter', age: 22}])
 * state
 * schema
 * exists?
-* alias_exists?
-* setting_exists?
-* mapping_exists?
-* meta_exists?
-
-Fast insert, update, delete raw data
-* index
-* insert
-* update
-* delete
-* bulk
+
+### plain methods
+* alias_exists?(...)
+* setting_exists?(...)
+* mapping_exists?(...)
+* meta_exists?(...)
+
+### Fast insert, update, delete raw data
+* index(...)
+* insert(...)
+* update(...)
+* delete(...)
+* bulk(...)
 
 -----
 
@@ -436,6 +469,9 @@ Access these methods through the model's connection or within any `Migration`.
 - create_table
 - change_table
 - rename_table
+- reindex_table
+- backup_table
+- restore_table
 
 ### table actions:
 - change_meta

data/docs/CHANGELOG.md

@@ -1,5 +1,31 @@
 # ElasticsearchRecord - CHANGELOG
 
+## [1.6.0] - 2023-08-11
+* [add] `ElasticsearchRecord::Base#undelegate_id_attribute_with` method to support a temporary 'undelegation' (used to create a new record)
+* [add] `ElasticsearchRecord::Relation#timeout` to directly provide the timeout-parameter to the query
+* [add] `ElasticsearchRecord.error_on_transaction`-flag to throw transactional errors (default: `false`) - this will now **IGNORE** all transactions
+* [add] `ElasticsearchRecord::ModelApi` create!, clone!, rename!, backup!, restore! & reindex!-methods
+* [add] `ElasticsearchRecord::Relation#pit_delete` which executes a delete query in a 'point_in_time' scope.
+* [add] `ActiveRecord::ConnectionAdapters::Elasticsearch::TableStatements#backup_table` to create a backup (snapshot) of the entire table (index)
+* [add] `ActiveRecord::ConnectionAdapters::Elasticsearch::TableStatements#restore_table` to restore a entire table (index)
+* [add] `ActiveRecord::ConnectionAdapters::Elasticsearch::TableStatements#reindex_table` to copy documents from source to destination
+* [ref] `ElasticsearchRecord::Base.delegate_id_attribute` now supports instance writer
+* [ref] `ElasticsearchRecord::Relation#pit_results` adds `ids_only`-parameter to now support a simple return of the records-ids...
+* [fix] Relation `#last`-method will raise an transport exception if cluster setting '**indices.id_field_data.enabled**' is disabled (now checks for `access_id_fielddata?`)
+* [fix] ElasticsearchRecord-connection settings does not support `username` key
+* [fix] ElasticsearchRecord-connection settings does not support `port` key
+* [fix] `_id`-Attribute is erroneously defined as 'virtual' attribute - but is required for insert statements.
+* [fix] unsupported **SAVEPOINT** transactions throws exceptions _(especially in tests)_
+* [fix] `ElasticsearchRecord::ModelApi#bulk` does not recognize `'_id' / :_id` attribute
+* [fix] `ElasticsearchRecord::ModelApi#bulk` does not correctly build the data-hash for `update`-operation _(missing 'doc'-node)_
+* [ref] simplify `ElasticsearchRecord::Base#searchable_column_names`
+* [fix] creating a new record does not recognize a manually provided `_id`-attribute
+* [fix] creating a new record with active `delegate_id_attribute`-flag does not update the records `_id`.
+
+## [1.5.3] - 2023-07-14
+* [fix] `ElasticsearchRecord::Relation#where!` on nested, provided `:none` key
+* [ref] minor code tweaks and comment updates
+
 ## [1.5.2] - 2023-07-12
 * [fix] `ElasticsearchRecord::Relation#limit` setter method `limit_value=` to work with **delegate_query_nil_limit?**
 
@@ -9,10 +35,10 @@
 
 ## [1.5.0] - 2023-07-10
 * [add] additional `ElasticsearchRecord::ModelApi` methods **drop!** & **truncate!**, which have to be called with a `confirm:true` parameter
-* [add] `.ElasticsearchRecord::Base.delegate_query_nil_limit` to automatically delegate a relations `limit(nil)`-call to the **max_result_window** _(set to 10.000 as default)_
+* [add] `ElasticsearchRecord::Base.delegate_query_nil_limit` to automatically delegate a relations `limit(nil)`-call to the **max_result_window** _(set to 10.000 as default)_
 * [add] `ActiveRecord::ConnectionAdapters::Elasticsearch::SchemaStatements#access_shard_doc?` which checks, if the **PIT**-shard_doc order is available
 * [add] support for **_shard_doc** as a default order for `ElasticsearchRecord::Relation#pit_results`
-* [ref] `.ElasticsearchRecord::Base.relay_id_attribute` to a more coherent name: `delegate_id_attribute` 
+* [ref] `ElasticsearchRecord::Base.relay_id_attribute` to a more coherent name: `delegate_id_attribute` 
 * [ref] `ElasticsearchRecord::Relation#ordered_relation` to optimize already ordered relations
 * [ref] gemspecs to support different versions of Elasticsearch
 * [ref] improved README

data/docs/CODE_OF_CONDUCT.md

@@ -39,7 +39,7 @@ This Code of Conduct applies within all community spaces, and also applies when
 
 ## Enforcement
 
-Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at tg@reimbursement.institute. All complaints will be reviewed and investigated promptly and fairly.
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at info@ruby-smart.org. All complaints will be reviewed and investigated promptly and fairly.
 
 All community leaders are obligated to respect the privacy and security of the reporter of any incident.
 

data/lib/active_record/connection_adapters/elasticsearch/table_statements.rb

@@ -116,7 +116,7 @@ module ActiveRecord
           #   Defaults to false.
           # @param [String] table_name
           # @param [Boolean] if_exists
-          # @return [Array] acknowledged status
+          # @return [Boolean] acknowledged status
           def drop_table(table_name, if_exists: false, **)
             schema_cache.clear_data_source_cache!(table_name)
             api(:indices, :delete, { index: table_name, ignore: (if_exists ? 404 : nil) }, 'DROP TABLE').dig('acknowledged')
@@ -148,11 +148,12 @@ module ActiveRecord
             end
           end
 
-          # clones an entire table (index) to the provided +target_name+.
+          # clones an entire table (index) with its docs to the provided +target_name+.
           # During cloning, the table will be automatically 'write'-blocked.
           # @param [String] table_name
           # @param [String] target_name
           # @param [Hash] options
+          # @return [Boolean] acknowledged status
           def clone_table(table_name, target_name, **options)
             # create new definition
             definition = clone_table_definition(table_name, target_name, **extract_table_options!(options))
@@ -168,6 +169,54 @@ module ActiveRecord
             definition.exec!
           end
 
+          # creates a backup (snapshot) of the entire table (index) from provided +table_name+.
+          # The backup will be closed, to prevent read/write access.
+          # The +target_name+ will be auto-generated, if not provided.
+          #
+          # @example
+          #   backup_table('screenshots', to: 'screenshots-backup-v1')
+          #
+          # @param [String] table_name
+          # @param [String] to - target_name
+          # @param [Boolean] close - closes backup after creation (default: true)
+          # @return [String] backup_name
+          def backup_table(table_name, to: nil, close: true)
+            to ||= "#{table_name}-snapshot-#{Time.now.strftime('%s%3N')}"
+            raise ArgumentError, "unable to backup '#{table_name}' to already existing target '#{to}'!" if table_exists?(to)
+
+            clone_table(table_name, to)
+            close_table(to) if close
+
+            to
+          end
+
+          # restores a entire table (index) from provided +target_name+.
+          # The +table_name+ will be dropped, if exists.
+          # The +from+ will persist, if not provided +drop_backup:true+.
+          #
+          # @example
+          #   restore_table('screenshots', from: 'screenshots-backup-v1')
+          #
+          # @param [String] table_name
+          # @param [String] from
+          # @param [String (frozen)] timeout - renaming timout (default: '30s')
+          # @param [Boolean] open - opens restored backup after creation (default: true)
+          # @return [Boolean] acknowledged status
+          def restore_table(table_name, from:, timeout: nil, open: true, drop_backup: false)
+            raise ArgumentError, "unable to restore from missing target '#{from}'!" unless table_exists?(from)
+            drop_table(table_name, if_exists: true)
+
+            # choose best strategy
+            if drop_backup
+              rename_table(from, table_name, timeout: timeout)
+            else
+              clone_table(from, table_name)
+            end
+
+            # open, if provided
+            open_table(from) if open
+          end
+
           # renames a table (index) by executing multiple steps:
           # - clone table
           # - wait for 'green' state
@@ -178,11 +227,11 @@ module ActiveRecord
           # @param [String] target_name
           # @param [String (frozen)] timeout (default: '30s')
           # @param [Hash] options - additional 'clone' options (like settings, alias, ...)
-          def rename_table(table_name, target_name, timeout: '30s', **options)
+          def rename_table(table_name, target_name, timeout: nil, **options)
             schema_cache.clear_data_source_cache!(table_name)
 
             clone_table(table_name, target_name, **options)
-            cluster_health(index: target_name, wait_for_status: 'green', timeout: timeout)
+            cluster_health(index: target_name, wait_for_status: 'green', timeout: timeout.presence || '30s')
             drop_table(table_name)
           end
 
@@ -255,6 +304,15 @@ module ActiveRecord
             definition.exec!
           end
 
+          # Copies documents from a source to a destination.
+          # @param [String] table_name
+          # @param [String] target_name
+          # @param [Hash] options
+          # @return [Hash] reindex stats
+          def reindex_table(table_name, target_name, **options)
+            api(:core, :reindex, { body: { source: { index: table_name }, dest: { index: target_name } } }.merge(options), 'REINDEX TABLE')
+          end
+
           # -- mapping -------------------------------------------------------------------------------------------------
 
           def add_mapping(table_name, name, type, **options, &block)

data/lib/active_record/connection_adapters/elasticsearch/transactions.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+module ActiveRecord
+  module ConnectionAdapters
+    module Elasticsearch
+      module Transactions
+        extend ActiveSupport::Concern
+
+        def transaction(*)
+          # since ActiveRecord does not have any configuration option to support transactions,
+          # this will be always false
+          # return super if supports_transactions?
+          #
+          # So, transactions are silently swallowed...
+          yield
+        end
+
+        # Begins the transaction (and turns off auto-committing).
+        def begin_db_transaction(*)
+          _throw_transaction_exception!(:begin_db_transaction)
+        end
+
+        # Commits the transaction (and turns on auto-committing).
+        def commit_db_transaction(*)
+          _throw_transaction_exception!(:commit_db_transaction)
+        end
+
+        # rollback transaction
+        def exec_rollback_db_transaction(*)
+          _throw_transaction_exception!(:exec_rollback_db_transaction)
+        end
+
+        def create_savepoint(*)
+          _throw_transaction_exception!(:create_savepoint)
+        end
+
+        def exec_rollback_to_savepoint(*)
+          _throw_transaction_exception!(:exec_rollback_to_savepoint)
+        end
+
+        def release_savepoint(*)
+          _throw_transaction_exception!(:release_savepoint)
+        end
+
+        private
+
+        def _throw_transaction_exception!(method_name)
+          return unless ElasticsearchRecord.error_on_transaction
+          raise NotImplementedError, "'##{method_name}' is not supported by Elasticsearch.\nTry to prevent transactions or set the 'ElasticsearchRecord.error_on_transaction' to false!"
+        end
+      end
+    end
+  end
+end

data/lib/active_record/connection_adapters/elasticsearch/unsupported_implementation.rb

@@ -3,13 +3,6 @@
 module ActiveRecord
   module ConnectionAdapters
     module Elasticsearch
-
-      class UnsupportedImplementationError < StandardError
-        def initialize(method_name)
-          super "Unsupported implementation of method: #{method_name}."
-        end
-      end
-
       module UnsupportedImplementation
         extend ActiveSupport::Concern
 

data/lib/active_record/connection_adapters/elasticsearch_adapter.rb

@@ -13,6 +13,7 @@ require 'active_record/connection_adapters/elasticsearch/schema_dumper'
 require 'active_record/connection_adapters/elasticsearch/schema_statements'
 require 'active_record/connection_adapters/elasticsearch/type'
 require 'active_record/connection_adapters/elasticsearch/table_statements'
+require 'active_record/connection_adapters/elasticsearch/transactions'
 
 require 'arel/visitors/elasticsearch'
 require 'arel/collectors/elasticsearch_query'
@@ -25,6 +26,12 @@ module ActiveRecord # :nodoc:
     def elasticsearch_connection(config)
       config          = config.symbolize_keys
 
+      # move 'username' to 'user'
+      config[:user]  = config.delete(:username) if config[:username]
+
+      # append 'port' to 'host'
+      config[:host]  += ":#{config.delete(:port)}" if config[:port] && config[:host]
+
       # move 'host' to 'hosts'
       config[:hosts]  = config.delete(:host) if config[:host]
 
@@ -45,7 +52,7 @@ module ActiveRecord # :nodoc:
 
       # defines the Elasticsearch 'base' structure, which is always included but cannot be resolved through mappings ...
       BASE_STRUCTURE = [
-        { 'name' => '_id', 'type' => 'keyword', 'virtual' => true, 'enabled' => true, 'meta' => { 'primary_key' => 'true' } },
+        { 'name' => '_id', 'type' => 'keyword', 'meta' => { 'primary_key' => 'true' } },
         { 'name' => '_index', 'type' => 'keyword', 'virtual' => true },
         { 'name' => '_score', 'type' => 'float', 'virtual' => true },
         { 'name' => '_type', 'type' => 'keyword', 'virtual' => true },
@@ -57,6 +64,7 @@ module ActiveRecord # :nodoc:
       include Elasticsearch::DatabaseStatements
       include Elasticsearch::SchemaStatements
       include Elasticsearch::TableStatements
+      include Elasticsearch::Transactions
 
       class << self
         def base_structure_keys
@@ -69,7 +77,7 @@ module ActiveRecord # :nodoc:
           client.ping unless config[:ping] == false
           client
         rescue ::Elastic::Transport::Transport::Errors::Unauthorized
-          raise ActiveRecord::DatabaseConnectionError.username_error(config[:username])
+          raise ActiveRecord::DatabaseConnectionError.username_error(config[:user])
         rescue ::Elastic::Transport::Transport::ServerError => error
           raise ::ActiveRecord::ConnectionNotEstablished, error.message
         end
@@ -135,7 +143,7 @@ module ActiveRecord # :nodoc:
 
       # define native types - which will be used for schema-dumping
       NATIVE_DATABASE_TYPES = {
-        primary_key: { name: 'long' },
+        primary_key: { name: 'long' }, # maybe this hae to changed to 'keyword'
         string:      { name: 'keyword' },
         blob:        { name: 'binary' },
         datetime:    { name: 'date' },
@@ -172,6 +180,12 @@ module ActiveRecord # :nodoc:
         @config[:migrations_paths] || ['db/migrate_elasticsearch']
       end
 
+      # Does this adapter support transactions in general?
+      # HINT: This is +NOT* an official setting and only introduced to ElasticsearchRecord
+      def supports_transactions?
+        false
+      end
+
       # Does this adapter support explain?
       def supports_explain?
         false

data/lib/arel/collectors/elasticsearch_query.rb

@@ -28,6 +28,9 @@ module Arel # :nodoc: all
         when :refresh
           # change the refresh state
           @refresh = args[0]
+        when :timeout
+          # change the timeout
+          @timeout = args[0]
         when :index
           # change the index name
           @index = args[0]

data/lib/arel/visitors/elasticsearch_query.rb

@@ -138,12 +138,11 @@ module Arel # :nodoc: all
 
       # CUSTOM node by elasticsearch_record
       def visit_Query(o)
-        # in some cases we don't have a kind, but where conditions.
-        # in this case we force the kind as +:bool+.
-        kind = :bool if o.wheres.present? && o.kind.blank?
-
-        # resolve kind, if not already set
-        kind ||= o.kind.present? ? visit(o.kind.expr) : nil
+        # resolves the query kind.
+        # PLEASE NOTE: in some cases there is no kind, but an existing +where+ conditions.
+        # This will then be treat as +:bool+.
+        kind = o.kind.present? ? visit(o.kind.expr).presence : nil
+        kind ||= :bool if o.wheres.present?
 
         # check for existing kind - we cannot create a node if we don't have any kind
         return unless kind
@@ -423,6 +422,7 @@ module Arel # :nodoc: all
         o
       end
 
+      # alias for RAW returns
       alias :visit_Integer :visit_Struct_Raw
       alias :visit_Symbol :visit_Struct_Raw
       alias :visit_Hash :visit_Struct_Raw
@@ -430,7 +430,6 @@ module Arel # :nodoc: all
       alias :visit_String :visit_Struct_Raw
       alias :visit_Arel_Nodes_SqlLiteral :visit_Struct_Raw
 
-
       # used by insert / update statements.
       # does not claim / assign any values!
       # returns a Hash of key => value pairs
@@ -453,6 +452,7 @@ module Arel # :nodoc: all
         o.name
       end
 
+      # alias for ATTRIBUTE returns
       alias :visit_Arel_Attributes_Attribute :visit_Struct_Attribute
       alias :visit_Arel_Nodes_UnqualifiedColumn :visit_Struct_Attribute
       alias :visit_ActiveModel_Attribute_FromUser :visit_Struct_Attribute
@@ -473,6 +473,7 @@ module Arel # :nodoc: all
         o.value
       end
 
+      # alias for BIND returns
       alias :visit_ActiveModel_Attribute :visit_Struct_BindValue
       alias :visit_ActiveRecord_Relation_QueryAttribute :visit_Struct_BindValue
 
@@ -484,6 +485,7 @@ module Arel # :nodoc: all
         collect(o)
       end
 
+      # alias for ARRAY returns
       alias :visit_Set :visit_Array
     end
   end

data/lib/arel/visitors/elasticsearch_schema.rb

@@ -89,8 +89,7 @@ module Arel # :nodoc: all
         # prepare query
         claim(:type, ::ElasticsearchRecord::Query::TYPE_INDEX_UPDATE_SETTING)
 
-        # special overcomplicated blocks to assign a hash of settings directly to the body...
-        # todo: refactor this in future versions
+        # overcomplicated blocks to assign a hash of settings directly to the body
         assign(:__query__, {}) do
           assign(:body, {}) do
             resolve(o.items, :visit_TableSettingDefinition)

data/lib/elasticsearch_record/core.rb

@@ -8,7 +8,7 @@ module ElasticsearchRecord
       # this through +_read_attribute(:id)+.
       # To also have the ability of accessing this attribute through the default, this flag can be enabled.
       # @attribute! Boolean
-      class_attribute :delegate_id_attribute, instance_writer: false, default: false
+      class_attribute :delegate_id_attribute, default: false
 
       # Elasticsearch's default value for queries without a +size+ is forced to +10+.
       # To provide a similar behaviour as SQL, this can be automatically set to the +max_result_window+ value.
@@ -45,7 +45,7 @@ module ElasticsearchRecord
 
     # overwrite to provide a Elasticsearch version of returning a 'primary_key' was attribute.
     # Elasticsearch uses the static +_id+ column as primary_key, but also supports an additional +id+ column.
-    # To provide functionality of returning the +id_Was+ attribute, this method must also support it
+    # To provide functionality of returning the +id_was+ attribute, this method must also support it
     # with enabled +delegate_id_attribute+.
     def id_was
       delegate_id_attribute? && has_attribute?('id') ? attribute_was('id') : super
@@ -69,6 +69,19 @@ module ElasticsearchRecord
       super
     end
 
+    # resets a possible active +delegate_id_attribute?+ to false during block execution.
+    # Unfortunately this is required, since a lot of rails-code forces 'accessors' on the primary_key-field through the
+    # +id+-getter & setter methods. This will then fail to set the doc-_id and instead set the +id+-attribute ...
+    def undelegate_id_attribute_with(&block)
+      return block.call unless self.delegate_id_attribute?
+
+      self.delegate_id_attribute = false
+      result = block.call
+      self.delegate_id_attribute = true
+
+      result
+    end
+
     module PrependClassMethods
       # returns the table_name.
       # Has to be prepended to provide automated compatibility to other gems.

data/lib/elasticsearch_record/gem_version.rb

@@ -8,8 +8,8 @@ module ElasticsearchRecord
 
   module VERSION
     MAJOR = 1
-    MINOR = 5
-    TINY  = 2
+    MINOR = 6
+    TINY  = 0
     PRE   = nil
 
     STRING = [MAJOR, MINOR, TINY, PRE].compact.join(".")

data/lib/elasticsearch_record/instrumentation/log_subscriber.rb

@@ -46,7 +46,7 @@ module ElasticsearchRecord
 
         # final coloring
         name  = color(name, name_color(payload[:name]), true)
-        query = color(query, gate_color(payload[:gate]), true) if colorize_logging
+        query = color(query, gate_color(payload[:gate], payload[:name]), true) if colorize_logging
 
         debug "  #{name} #{query.presence || '-/-'}"
       end
@@ -61,7 +61,7 @@ module ElasticsearchRecord
         end
       end
 
-      def gate_color(gate)
+      def gate_color(gate, name)
         case gate
           # SELECTS
         when 'core.get', 'core.mget', 'core.search', 'core.msearch', 'core.count', 'core.exists', 'sql.query'
@@ -77,7 +77,11 @@ module ElasticsearchRecord
           YELLOW
           # MIXINS
         when /indices\.\w+/, 'core.bulk', 'core.index'
-          WHITE
+          if name.end_with?('Pit Delete')
+            RED
+          else
+            WHITE
+          end
         else
           MAGENTA
         end

data/lib/elasticsearch_record/model_api.rb

@@ -8,9 +8,6 @@ module ElasticsearchRecord
       @klass = klass
     end
 
-    # undelegated schema methods: clone rename create
-    # those should not be quick-accessible, since they might end in heavily broken index
-
     # delegated dangerous methods (created with exclamation mark)
     # not able to provide individual arguments - always the defaults will be used!
     #
@@ -26,6 +23,21 @@ module ElasticsearchRecord
       end
     end
 
+    # delegated dangerous methods with args
+    #
+    # @example
+    #   create!(:new_table_name, settings: , mappings:, alias: , ...)
+    #   clone!(:new_table_name)
+    #   rename!(:new_table_name)
+    #   backup!(to: :backup_name)
+    #   restore!(from: :backup_name)
+    #   reindex!(:new_table_name)
+    %w(create clone rename backup restore reindex).each do |method|
+      define_method("#{method}!") do |*args|
+        _connection.send("#{method}_table", _index_name, *args)
+      end
+    end
+
     # delegated dangerous methods with confirm parameter (created with exclamation mark)
     # a exception will be raised, if +confirm:true+ is missing.
     #
@@ -146,12 +158,51 @@ module ElasticsearchRecord
     # Shortcut for meta_exists
     # @return [Boolean]
 
+    # @!method create!(force: false, copy_from: nil, if_not_exists: false, **options)
+    # Shortcut for create_table
+    # @param [Boolean] force
+    # @param [nil, String] copy_from
+    # @param [Hash] options
+    # @return [Boolean] acknowledged status
+
+    # @!method clone!(target_name, **options)
+    # Shortcut for clone_table
+    # @param [String] target_name
+    # @param [Hash] options
+    # @return [Boolean]
+
+    # @!method rename!(target_name, timeout: nil, **options)
+    # Shortcut for rename_table
+    # @param [String] target_name
+    # @param [String (frozen)] timeout
+    # @param [Hash] options
+
+    # @!method backup!(to: nil, close: true)
+    # Shortcut for backup_table
+    # @param [String] to
+    # @param [Boolean] close
+    # @return [String] backup_name
+
+    # @!method restore!(from:, timeout: nil, open: true, drop_backup: false)
+    # Shortcut for restore_table
+    # @param [String] from
+    # @param [String (frozen)] timeout
+    # @param [Boolean] open
+    # @return [Boolean] acknowledged status
+
+    # @!method reindex!(target_name, **options)
+    # Shortcut for reindex_table
+    # @param [String] target_name
+    # @param [Hash] options
+    # @return [Hash] reindex stats
+
     # fast insert/update data.
+    # IMPORTANT: Any 'doc'-id must by provided with underscore '_' ( +:_id+ )
     #
     # @example
     #   index([{name: 'Hans', age: 34}, {name: 'Peter', age: 22}])
     #
-    #   index({id: 5, name: 'Georg', age: 87})
+    #   index({_id: 5, name: 'Georg', age: 87})
     #
     # @param [Array<Hash>,Hash] data
     # @param [Hash] options
@@ -160,6 +211,7 @@ module ElasticsearchRecord
     end
 
     # fast insert new data.
+    # IMPORTANT: Any 'doc'-id must by provided with underscore '_' ( +:_id+ )
     #
     # @example
     #   insert([{name: 'Hans', age: 34}, {name: 'Peter', age: 22}])
@@ -173,11 +225,12 @@ module ElasticsearchRecord
     end
 
     # fast update existing data.
+    # IMPORTANT: Any 'doc'-id must by provided with underscore '_' ( +:_id+ )
     #
     # @example
-    #   update([{id: 1, name: 'Hansi'}, {id: 2, name: 'Peter Parker', age: 42}])
+    #   update([{_id: 1, name: 'Hansi'}, {_id: 2, name: 'Peter Parker', age: 42}])
     #
-    #   update({id: 3, name: 'Georg McCain'})
+    #   update({_id: 3, name: 'Georg McCain'})
     #
     # @param [Array<Hash>,Hash] data
     # @param [Hash] options
@@ -186,13 +239,14 @@ module ElasticsearchRecord
     end
 
     # fast delete data.
+    # IMPORTANT: Any 'doc'-id must by provided with underscore '_' ( +:_id+ )
     #
     # @example
     #   delete([1,2,3,5])
     #
     #   delete(3)
     #
-    #   delete({id: 2})
+    #   delete({_id: 2})
     #
     # @param [Array<Hash>,Hash] data
     # @param [Hash] options
@@ -202,12 +256,12 @@ module ElasticsearchRecord
       if data[0].is_a?(Hash)
         bulk(data, :delete, **options)
       else
-        bulk(data.map { |id| { id: id } }, :delete, **options)
+        bulk(data.map { |id| { _id: id } }, :delete, **options)
       end
     end
 
     # bulk handle provided data (single Hash or multiple Array<Hash>).
-    # @param [Hash,Array<Hash>] data - the data to insert/update/delete ...
+    # @param [Hash,Array<Hash<Symbol=>Object>>] data - the data to insert/update/delete ...
     # @param [Symbol] operation
     # @param [Boolean, Symbol] refresh
     def bulk(data, operation = :index, refresh: true, **options)
@@ -215,7 +269,11 @@ module ElasticsearchRecord
 
       _connection.api(:core, :bulk, {
         index:   _index_name,
-        body:    data.map { |item| { operation => { _id: item[:id], data: item.except(:id) } } },
+        body:    if operation == :update
+                   data.map { |item| { operation => { _id: (item[:_id].presence || item['_id']), data: { doc: item.except(:_id, '_id') } } } }
+                 else
+                   data.map { |item| { operation => { _id: (item[:_id].presence || item['_id']), data: item.except(:_id, '_id') } } }
+                 end,
         refresh: refresh
       }, "BULK #{operation.to_s.upcase}", **options)
     end

data/lib/elasticsearch_record/model_schema.rb

@@ -52,11 +52,8 @@ module ElasticsearchRecord
       # @return [Array<String>]
       def searchable_column_names
         @searchable_column_names ||= columns.select(&:enabled?).reduce([]) { |m, column|
-          m << column.name
-          m += column.field_names
-          m += column.property_names
-          m.uniq
-        }
+          m + [column.name] + column.field_names + column.property_names
+        }.uniq
       end
 
       # clears schema-related instance variables.

data/lib/elasticsearch_record/persistence.rb

@@ -11,7 +11,7 @@ module ElasticsearchRecord
         # values is not a "key=>values"-Hash, but a +ActiveModel::Attribute+ - so the casted values gets resolved here
         values = values.transform_values(&:value)
 
-        # resolve & update a auto_increment value
+        # resolve & update a auto_increment value, if configured
         _insert_with_auto_increment(values) do |arguments|
           # build new query
           query = ElasticsearchRecord::Query.new(
@@ -61,21 +61,20 @@ module ElasticsearchRecord
 
       private
 
-      # WARNING: BETA!!!
       # Resolves the +auto_increment+ status from the tables +_meta+ attributes.
       def _insert_with_auto_increment(values)
-        # check, if the primary_key's values is provided.
-        # so, no need to resolve a +auto_increment+ value, but provide
-        if values[self.primary_key].present?
-          # resolve id from values
-          id = values[self.primary_key]
-
+        # check, if the primary_key's value is provided.
+        # so, no need to resolve a +auto_increment+ value, but provide the id directly
+        if (id = values[self.primary_key]).present?
           yield({id: id})
         elsif auto_increment?
+          # future increments: uuid (+uuidv6 ?), hex, radix(2-36), integer
+          # allocated through: primary_key_type
+
           ids = [
-            # we try to resolve the current-auto-increment value from the tables meta
+            # try to resolve the current-auto-increment value from the tables meta
             connection.table_metas(self.table_name).dig('auto_increment').to_i + 1,
-            # for secure reasons, we also resolve the current maximum value for the primary key
+            # for secure reasons: also resolve the current maximum value for the primary key
             self.unscoped.all.maximum(self.primary_key).to_i + 1
           ]
 
@@ -92,5 +91,14 @@ module ElasticsearchRecord
         end
       end
     end
+
+    # overwrite to provide a Elasticsearch version:
+    # Creates a record with values matching those of the instance attributes
+    # and returns its id.
+    def _create_record(*args)
+      undelegate_id_attribute_with do
+        super
+      end
+    end
   end
 end

data/lib/elasticsearch_record/query.rb

@@ -86,9 +86,9 @@ module ElasticsearchRecord
     # @!attribute Boolean
     attr_reader :refresh
 
-    # defines the query body - in most cases this is a hash
-    # @!attribute Hash
-    # attr_reader :body
+    # defines the query timeout
+    # @!attribute Integer|String
+    attr_reader :timeout
 
     # defines the query arguments to be passed to the API
     # @!attribute Hash
@@ -98,11 +98,12 @@ module ElasticsearchRecord
     # @!attribute Array
     attr_reader :columns
 
-    def initialize(index: nil, type: TYPE_UNDEFINED, status: STATUS_VALID, body: nil, refresh: nil, arguments: {}, columns: [])
+    def initialize(index: nil, type: TYPE_UNDEFINED, status: STATUS_VALID, body: nil, refresh: nil, timeout: nil, arguments: {}, columns: [])
       @index     = index
       @type      = type
       @status    = status
       @refresh   = refresh
+      @timeout   = timeout
       @body      = body
       @arguments = arguments
       @columns   = columns
@@ -163,6 +164,9 @@ module ElasticsearchRecord
       # set refresh, if defined (also includes false value)
       args[:refresh] = self.refresh unless self.refresh.nil?
 
+      # set timeout, if present
+      args[:timeout] = self.timeout if self.timeout.present?
+
       args
     end
 

data/lib/elasticsearch_record/relation/core_methods.rb

@@ -125,6 +125,18 @@ module ElasticsearchRecord
           self
         end
       end
+
+      # overwrite original methods to provide a elasticsearch version:
+      # checks against the +#access_id_fielddata?+ to ensure the Elasticsearch Cluster allows access on the +_id+ field.
+      def reverse_sql_order(order_query)
+        if order_query.empty?
+          return [table[primary_key].desc] if primary_key != '_id' || klass.connection.access_id_fielddata?
+          raise ActiveRecord::IrreversibleOrderError,
+                "Relation has no current order and fielddata access on the _id field is disallowed! However, you can re-enable it by updating the dynamic cluster setting: indices.id_field_data.enabled"
+        end
+
+        super
+      end
     end
   end
 end

data/lib/elasticsearch_record/relation/query_methods.rb

@@ -102,6 +102,16 @@ module ElasticsearchRecord
         configure!(:__query__, refresh: value)
       end
 
+      # sets the query's +timeout+ value.
+      # @param [Boolean] value (default: true)
+      def timeout(value = true)
+        spawn.timeout!(value)
+      end
+
+      def timeout!(value = true)
+        configure!(:__query__, timeout: value)
+      end
+
       # add a whole query 'node' to the query.
       # @example
       #   query(:bool, {filter: ...})
@@ -184,35 +194,36 @@ module ElasticsearchRecord
 
       # creates a condition on the relation.
       # There are several possibilities to call this method.
+      #
       # @example
       #   # create a simple 'term' condition on the query[:filter] param
-      #     where({name: 'hans'})
-      #     > query[:filter] << { term: { name: 'hans' } }
+      #   where({name: 'hans'})
+      #   #> query[:filter] << { term: { name: 'hans' } }
       #
       #   # create a simple 'terms' condition on the query[:filter] param
-      #     where({name: ['hans','peter']})
-      #     > query[:filter] << { terms: { name: ['hans','peter'] } }
+      #   where({name: ['hans','peter']})
+      #   #> query[:filter] << { terms: { name: ['hans','peter'] } }
       #
-      #     where(:must_not, term: {name: 'horst'})
-      #     where(:query_string, "(new york OR dublin)", fields: ['name','description'])
+      #   where(:must_not, term: {name: 'horst'})
+      #   where(:query_string, "(new york OR dublin)", fields: ['name','description'])
       #
       #   # nested array
       #   where([ [:filter, {...}], [:must_not, {...}]])
-      def where(*args)
-        return none if args[0] == :none
-
-        super
-      end
-
+      #
+      #   # invalidate query
+      #   where(:none)
+      #
       def where!(opts, *rest)
         # :nodoc:
         case opts
-          # check the first provided parameter +opts+ and validate, if this is an alias for "must, must_not, should or filter"
-          # if true, we expect the rest[0] to be a hash.
-          # For this correlation we forward this as RAW-data without check & manipulation
         when Symbol
           case opts
+          when :none
+            none!
           when :filter, :must, :must_not, :should
+            # check the first provided parameter +opts+ and validate, if this is an alias for "must, must_not, should or filter".
+            # if true, we expect the rest[0] to be a hash.
+            # For this correlation we forward this as RAW-data without check & manipulation
             send("#{opts}!", *rest)
           else
             raise ArgumentError, "Unsupported prefix type '#{opts}'. Allowed types are: :filter, :must, :must_not, :should"
@@ -220,9 +231,7 @@ module ElasticsearchRecord
         when Array
           # check if this is a nested array of multiple [<kind>,<data>]
           if opts[0].is_a?(Array)
-            opts.each { |item|
-              where!(*item)
-            }
+            opts.each { |item| where!(*item) }
           else
             where!(*opts, *rest)
           end

data/lib/elasticsearch_record/relation/result_methods.rb

@@ -90,7 +90,9 @@ module ElasticsearchRecord
       #
       # @param [String] keep_alive - how long to keep alive (for each single request) - default: '1m'
       # @param [Integer] batch_size - how many results per query (default: 1000 - this means at least 10 queries before reaching the +max_result_window+)
-      def pit_results(keep_alive: '1m', batch_size: 1000)
+      # @param [Boolean] ids_only - resolve ids only from results
+      # @return [Integer, Array] either returns the results-array (no block provided) or the total amount of results
+      def pit_results(keep_alive: '1m', batch_size: 1000, ids_only: false)
         raise(ArgumentError, "Batch size cannot be above the 'max_result_window' (#{klass.max_result_window}) !") if batch_size > klass.max_result_window
 
         # check if limit or offset values where provided
@@ -105,6 +107,9 @@ module ElasticsearchRecord
         # see @ https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html
         relation.order!(_shard_doc: :asc) if relation.order_values.empty? && klass.connection.access_shard_doc?
 
+        # resolve ids only
+        relation.reselect!('_id') if ids_only
+
         # clear limit & offset
         relation.offset!(nil).limit!(nil)
 
@@ -122,10 +127,16 @@ module ElasticsearchRecord
           # resolve new data until we got all we need
           loop do
             # change pit settings & limit (spawn is required, since a +resolve+ will make the relation immutable)
-            current_response = relation.spawn.configure!(current_pit_hash).limit!(batch_size).resolve('Pit').response
+            current_response = relation.spawn.configure!(current_pit_hash).limit!(batch_size).resolve('Pit Results').response
 
             # resolve only data from hits->hits[{_source}]
-            current_results        = current_response['hits']['hits'].map { |result| result['_source'].merge('_id' => result['_id']) }
+            current_results        = if ids_only
+                                       current_response['hits']['hits'].map { |result| result['_id'] }
+                                       # future with helper
+                                       # current_response['hits']['hits'].map.from_hash('_id')
+                                     else
+                                       current_response['hits']['hits'].map { |result| result['_source'].merge('_id' => result['_id']) }
+                                     end
             current_results_length = current_results.length
 
             # check if we reached the required offset
@@ -171,12 +182,38 @@ module ElasticsearchRecord
           end
         end
 
-        # return results array
-        results
+        # return results array or total value
+        if block_given?
+          results_total
+        else
+          results
+        end
       end
 
       alias_method :total_results, :pit_results
 
+      # executes a delete query in a +point_in_time+ scope.
+      # this will provide the possibility to delete more than the +max_result_window+ (default: 10000) docs in a batched process.
+      # @param [String] keep_alive
+      # @param [Integer] batch_size
+      # @param [Boolean] refresh index after delete finished (default: true)
+      # @return [Integer] total amount of deleted docs
+      def pit_delete(keep_alive: '1m', batch_size: 1000, refresh: true)
+        delete_count = select('_id').pit_results(keep_alive: keep_alive, batch_size: batch_size, ids_only: true) do |ids|
+          # skip empty results
+          next unless ids.any?
+
+          # delete all IDs, but do not refresh index, yet
+          klass.connection.api(:core, :bulk, { index: klass.table_name, body: ids.map { |id| { delete: { _id: id } } }, refresh: false }, "#{klass} Pit Delete")
+        end
+
+        # refresh index
+        klass.connection.refresh_table(klass.table_name) if refresh
+
+        # return total count
+        delete_count
+      end
+
       # returns the RAW response for the current query
       # @return [Array]
       def response

data/lib/elasticsearch_record/result.rb

@@ -49,7 +49,7 @@ module ElasticsearchRecord
     end
 
     # Returns the RAW +_source+ data from each hit - aka. +rows+.
-    # PLEASE NOTE: The array will only contain the RAW data from each +_source+ (meta info like '_score' is not included)
+    # PLEASE NOTE: The array will only contain the RAW data from each +_source+ (meta info like '_id' or '_score' are not included)
     # @return [Array]
     def results
       return [] unless response['hits']

data/lib/elasticsearch_record.rb

@@ -55,6 +55,16 @@ module ElasticsearchRecord
 
     autoload :ElasticsearchDatabaseTasks, 'elasticsearch_record/tasks/elasticsearch_database_tasks'
   end
+
+  ##
+  # :singleton-method:
+  # Specifies if a exception should be raised while using transactions.
+  # Since ActiveRecord does not have any configuration option to support transactions and
+  # Elasticsearch does **NOT** support transactions, it may be risky to ignore them.
+  # As default, transactional are 'silently swallowed' to not break any existing applications...
+  # However enabling this flag will surely fail transactional tests ...
+  singleton_class.attr_accessor :error_on_transaction
+  self.error_on_transaction = false
 end
 
 ActiveSupport.on_load(:active_record) do

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: elasticsearch_record
 version: !ruby/object:Gem::Version
-  version: 1.5.2
+  version: 1.6.0
 platform: ruby
 authors:
 - Tobias Gonsior
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2023-07-12 00:00:00.000000000 Z
+date: 2023-08-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -145,6 +145,7 @@ files:
 - lib/active_record/connection_adapters/elasticsearch/schema_dumper.rb
 - lib/active_record/connection_adapters/elasticsearch/schema_statements.rb
 - lib/active_record/connection_adapters/elasticsearch/table_statements.rb
+- lib/active_record/connection_adapters/elasticsearch/transactions.rb
 - lib/active_record/connection_adapters/elasticsearch/type.rb
 - lib/active_record/connection_adapters/elasticsearch/type/format_string.rb
 - lib/active_record/connection_adapters/elasticsearch/type/multicast_value.rb