logstash-input-elastic_jdbc 0.1.4 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: eb430d13b3d23a0c91d2a550aa65efc1beef6a4ff7dfdd726192ebb39c8427e1
-  data.tar.gz: b443c0f66c569b9b3c2ce3dd9ab5d18cff46c7ec7601bbf0211a18fd8addb541
+  metadata.gz: 956e0978c2ea4e70c4b8410a90fc610113f3c46c12afdec85014ce6ffcd2d1ae
+  data.tar.gz: c191601527d553ba277c4383122ba053c5fb02a836f77416b65b117260a36cab
 SHA512:
-  metadata.gz: d0ffd506739f0cf07b6536ca137aaa9b46f54ac74a8a9e86fd048447760371cb0701ab0eaea8729f258ea82ef7165b9446d86fa4dc28d18b3907f7b5865457b8
-  data.tar.gz: 9c7f4e71931d00c2fd5d8ca46893290dd4cf2a57d9ddf6b82fc63ac3015814db949d29dbca2971136a1742a1913ed51afd88c49c2397332e3701270bc4d7e9bd
+  metadata.gz: d89379f8a8a87ffde484cd105b1fb9fe3760498375ee3b8bf90e230da185feaeea199a4b942b74e61049035ad8ad076898aecd9f9e2a95ab9a61a90107a2d42e
+  data.tar.gz: ba304b692417cd56fdc4aba032c4a58be96033b44bd2b522aadf531e6156b3b64c5d95f388bb3912b7079283fdf571690b14fcb6fed7571d812d9d5c35b2cc8a

data/README.md

@@ -1,33 +1,66 @@
 # Logstash Plugin
-[GitHub](https://github.com/ernesrocker/RubyGems).
+[GitHub](https://github.com/ernesrocker/logstash-input-elastic_jdbc).
 This is a plugin for [Logstash](https://github.com/elastic/logstash).
 
 It is fully free and fully open source.
 
+## How install
+```sh
+sudo /usr/share/logstash/bin/logstash-plugin install logstash-input-elastic_jdbc.gem
+```
+
 ## Documentation
-This plugin inherit of elasticsearch input plugin, and added a tracking_column
+This plugin inherit of elasticsearch(**ES**) input plugin, and added a tracking_column
 using in jdbc input plugin for make a query to obtain the updates values
 Sample :
- input {
-       # Read all documents from Elasticsearch matching the given query
-       elastic_jdbc {
+```logstash
+  input{
+       elastic_jdbc{
          hosts => "localhost"
+         index => "documents"
          tracking_column => "last_update"
- 	            last_run_metadata_path => "/path_file"
- 	    }
-     }
-
-Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
-
-- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
-- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
-
-## Need Help?
-
-Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
-
-## Developing
-
+         query => '{"query":{"range":{"created":{"gte":"2021-08-13T00:17:58+00:00"}}}}'
+         last_run_metadata_path => "/opt/logstash/last_run/elastic_jdbc_documents"
+       }
+  }
+  filter {
+  }
+  output{
+    stdout{}
+  }
+```  
+In the sample before, we read from ES cluster, **documents** index, where documents hits have last_update field as 
+a **date** type field (recommend use [Ingest pipelines](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/ingest.html)),
+then we look for all documents that have a field value **last_update** greater than the value stored in `/opt/logstash/last_run/elastic jdbc_documents" `.
+
+####Required parameters:
+   * `hosts`: ES cluster url
+   * `index`: ES index
+   * `tracking_column`: Date field to tracking in ES index
+   * `last_run_metadata_path` : File path where stored the last value from last hist readed from ES index. By the default have the date `1960-01-01`
+
+####Optional parameters:
+   * All [logstash-input-elasticsearch](https://rubygems.org/gems/logstash-input-elasticsearch) parameters can use in this plugins.
+   * `query`: By the default we use a bool query where we get a hits with `tracking column` greater that last value stored in `last_run_metadata_path`. 
+   you can insert a query, but keep in mind that your query always be appended with the default query ( *if you don't need search by tracking column,
+   please use [logstash-input-elasticsearch](https://rubygems.org/gems/logstash-input-elasticsearch) plugin*). 
+   Sample, for this query parameter ``query => '{"query":{"range":{"created":{"gte":"2021-08-13T00:17:58+00:00"}}}}'``, 
+   the final query using this plugin would be:
+    
+   ```{
+       "query":{
+          "bool":{
+             "must":[
+                  {"range": {"last_update":{"gt": "date_time_value_stored"}}},
+                  {"range":{"abonado_date":{"gte": "2021-08-13T00:17:58+00:00"}}}
+             ]
+          }
+       }, 
+       sort: [{"last_update"=>{:order=>"asc"}}]
+      }
+   ```
+   **Note:** If you insert a ranking attribute within the query, we always overwrite it with the ranking values shown above.
+    
 ### 1. Plugin Developement and Testing
 
 #### Code

data/lib/logstash/inputs/elastic_jdbc.rb

@@ -1,8 +1,7 @@
 # encoding: utf-8
 require "logstash/inputs/base"
 require "logstash/inputs/elasticsearch"
-require_relative "value_tracking"
-require "json"
+require_relative "../../logstash/inputs/value_tracking"
 require "logstash/json"
 require "time"
 
@@ -23,6 +22,9 @@ require "time"
 class LogStash::Inputs::ElasticJdbc < LogStash::Inputs::Elasticsearch
   config_name "elastic_jdbc"
 
+  # Overwrite query default of elasticsearch plugin. We build a default query in this plugins.
+  config :query, :validate => :string, :default => '{}'
+
   #region tracking configuration
   # Path to file with last run time
   config :last_run_metadata_path, :validate => :string, :default => "#{ENV['HOME']}/.logstash_jdbc_last_run"
@@ -43,16 +45,12 @@ class LogStash::Inputs::ElasticJdbc < LogStash::Inputs::Elasticsearch
 
   public
   def register
-    super
-    begin
-      if @tracking_column.nil?
-        raise(LogStash::ConfigurationError, "Must set :tracking_column if :use_column_value is true.")
-      end
-      set_value_tracker(ValueTracking.build_last_value_tracker(self))
-      build_query()
-    rescue
-      puts "[ERROR:ELASTIC_JDBC]: #{e}"
+    if @tracking_column.nil?
+      raise(LogStash::ConfigurationError, "Must set :tracking_column if :use_column_value is true.")
     end
+    @value_tracker = ValueTracking.build_last_value_tracker(self)
+    super
+    build_query
   end # def register
 
   def set_value_tracker(instance)
@@ -60,25 +58,79 @@ class LogStash::Inputs::ElasticJdbc < LogStash::Inputs::Elasticsearch
   end
 
   def build_query
+    input_query = @base_query
+    # Remove sort tag from base query. We only sort by tracking column
+    input_query.delete("sort")
     time_now = Time.now.utc
     last_value = @value_tracker ? Time.parse(@value_tracker.value.to_s).iso8601 : Time.parse(time_now).iso8601
     column = @tracking_column.to_s
-    query = {query: { range: {column => {gt: last_value.to_s}}}, sort: [{column => {order: "desc"}}]}
-    @query = query.to_json
-    @base_query = LogStash::Json.load(@query)
+    query_default = {query: { bool: { must: [ {range: {column => {gt: last_value.to_s}}} ]}}}
+    if !input_query.nil? and !input_query.empty?
+      query_conditions = input_query["query"] 
+      if query_conditions
+        must_statement = query_default[:query][:bool][:must]
+        final_must_cond = must_statement.append(query_conditions)
+        query_default[:query][:bool][:must] = final_must_cond
+      end
+    end
+    sort_condition = [{column => {order: "asc"}}]
+    query_default[:sort] = sort_condition
+    @base_query = LogStash::Json.load(query_default.to_json)
   end
 
   def run(output_queue)
-    begin
-      super
-      @value_tracker.set_value(Time.now.to_s)
-      @value_tracker.write
-    rescue Exception =>e
-      puts "[ERROR:ELASTIC_JDBC]: #{e.message}"
-      puts "Full exception: #{e}"
-    end
+    super
   end # def run
 
+  def do_run_slice(output_queue, slice_id=nil)
+    slice_query = @base_query
+    slice_query = slice_query.merge('slice' => { 'id' => slice_id, 'max' => @slices}) unless slice_id.nil?
+
+    slice_options = @options.merge(:body => LogStash::Json.dump(slice_query) )
+    logger.info("Slice starting", slice_id: slice_id, slices: @slices) unless slice_id.nil?
+    r = search_request(slice_options)
+
+    r['hits']['hits'].each { |hit| push_hit(hit, output_queue) }
+    logger.debug("Slice progress", slice_id: slice_id, slices: @slices) unless slice_id.nil?
+
+    has_hits = r['hits']['hits'].any?
+
+    while has_hits && r['_scroll_id'] && !stop?
+      r = process_next_scroll(output_queue, r['_scroll_id'])
+      logger.debug("Slice progress", slice_id: slice_id, slices: @slices) unless slice_id.nil?
+      has_hits = r['has_hits']
+    end
+    logger.info("Slice complete", slice_id: slice_id, slices: @slices) unless slice_id.nil?
+  end
+
+  def push_hit(hit, output_queue)
+    event = LogStash::Event.new(hit['_source'])
+
+    if @docinfo
+      # do not assume event[@docinfo_target] to be in-place updatable. first get it, update it, then at the end set it in the event.
+      docinfo_target = event.get(@docinfo_target) || {}
+
+      unless docinfo_target.is_a?(Hash)
+        @logger.error("Elasticsearch Input: Incompatible Event, incompatible type for the docinfo_target=#{@docinfo_target} field in the `_source` document, expected a hash got:", :docinfo_target_type => docinfo_target.class, :event => event)
+
+        # TODO: (colin) I am not sure raising is a good strategy here?
+        raise Exception.new("Elasticsearch input: incompatible event")
+      end
+
+      @docinfo_fields.each do |field|
+        docinfo_target[field] = hit[field]
+      end
+
+      event.set(@docinfo_target, docinfo_target)
+    end
+
+    decorate(event)
+    output_queue << event
+    # Write in the file the last_update value register in the event.
+    @value_tracker.set_value(event.get(@tracking_column))
+    @value_tracker.write
+  end
+
   def stop
     super
   end

data/logstash-input-elastic-jdbc.gemspec

@@ -1,10 +1,12 @@
+# coding: utf-8
 Gem::Specification.new do |s|
   s.name          = 'logstash-input-elastic_jdbc'
-  s.version       = '0.1.4'
+  s.version       = '1.0.0'
   s.licenses      = ['Apache-2.0']
-  s.summary       = 'Logstash elastic_jdbc'
-  s.description   = 'This plugin inherit of elasticsearch input plugin, but added tracking_column like jdbc input plugin.'
-  s.homepage      = 'https://github.com/ernesrocker/RubyGems'
+  s.summary       = 'Reads querys from Elasticsearch cluster and write last run file.'
+  s.description   = 'This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname.
+                     This gem is not a stand-alone program. Also, this plugin inherit of elasticsearch input plugin, but added tracking_column like jdbc input plugin.'
+  s.homepage      = 'https://github.com/ernesrocker/logstash-input-elastic_jdbc'
   s.authors       = ['Ernesto Soler Calaña']
   s.email         = 'ernes920825@gmail.com'
   s.require_paths = ['lib']
@@ -22,4 +24,5 @@ Gem::Specification.new do |s|
   s.add_runtime_dependency 'logstash-codec-plain'
   s.add_runtime_dependency 'stud', '>= 0.0.22'
   s.add_development_dependency 'logstash-devutils', '~> 0.0', '>= 0.0.16'
+  s.add_development_dependency 'logstash-input-elasticsearch', '>= 4.3.1'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-input-elastic_jdbc
 version: !ruby/object:Gem::Version
-  version: 0.1.4
+  version: 1.0.0
 platform: ruby
 authors:
 - Ernesto Soler Calaña
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-11-06 00:00:00.000000000 Z
+date: 2021-09-13 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: logstash-core-plugin-api
@@ -72,8 +72,23 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 0.0.16
-description: This plugin inherit of elasticsearch input plugin, but added tracking_column
-  like jdbc input plugin.
+- !ruby/object:Gem::Dependency
+  name: logstash-input-elasticsearch
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 4.3.1
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 4.3.1
+description: |-
+  This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname.
+                       This gem is not a stand-alone program. Also, this plugin inherit of elasticsearch input plugin, but added tracking_column like jdbc input plugin.
 email: ernes920825@gmail.com
 executables: []
 extensions: []
@@ -89,7 +104,7 @@ files:
 - lib/logstash/inputs/value_tracking.rb
 - logstash-input-elastic-jdbc.gemspec
 - spec/inputs/elastic-jdbc_spec.rb
-homepage: https://github.com/ernesrocker/RubyGems
+homepage: https://github.com/ernesrocker/logstash-input-elastic_jdbc
 licenses:
 - Apache-2.0
 metadata:
@@ -114,6 +129,6 @@ rubyforge_project:
 rubygems_version: 2.7.6
 signing_key: 
 specification_version: 4
-summary: Logstash elastic_jdbc
+summary: Reads querys from Elasticsearch cluster and write last run file.
 test_files:
 - spec/inputs/elastic-jdbc_spec.rb