logstash-input-rethinker 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 5026e50d97fea101dc31c6023a563c84456b492a
+  data.tar.gz: 3924c222de9b343df2405c7b3c9ec0b35c2a79fc
+SHA512:
+  metadata.gz: 71158b58a616f62f6aa7487977159ce5c67407876a6e2f3d422b4a1ee36d8b9050f9c849e12b437f6d01d05c5a73bb9031894b235c11867d2f0ca0684d2c5955
+  data.tar.gz: 4e540e573ef48cd56876fc510be71ad332a77759e13442b472594aa33f85f17d59ab0594592fda1b89e62713f05a1cbc5506dc663864223f611eb2f60219c55a

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+## 0.1.0
+  - Plugin created with the logstash plugin generator

data/CONTRIBUTORS

@@ -0,0 +1,10 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Wayann - wayann@nrg.re
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,2 @@
+# logstash-input-rethinker
+Example input plugin. This should help bootstrap your effort to write your own input plugin!

data/Gemfile

@@ -0,0 +1,5 @@
+source 'https://rubygems.org'
+gem "eventmachine"
+gem "rethinkdb"
+gemspec
+

data/LICENSE

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,50 @@
+# Rethinker Logstash input plugin
+
+This is a plugin for [Logstash](https://github.com/elasticsearch/logstash).
+
+**Currently this is BETA software. It contains some known limitations (see below)**
+
+## Using the plugin
+
+You'll need to use this with the Logstash 1.5 or higher, which you can [download here](https://www.elastic.co/downloads/logstash).
+
+- Install the plugin from the Logstash home directory (you will need jruby to bundle install)
+```sh
+$ git clone https://github.com/wayann/logstash-input-rethinker
+$ cd logstash-input-rethinker
+$ bundle install
+$ gem build logstash-input-rethinker.gemspec
+$ bin/plugin install --no-verify logstash-input-rethinker-0.1.0.gem
+```
+
+- Now you can test the plugin using the stdout output:
+
+```sh
+$ bin/logstash -f rethinker.conf
+```
+
+This will immediately watch the tables `test.foo` and `db2.baz`, and it will also watch the databases `db1` and `db2` for new or dropped tables and watch or unwatch those tables appropriately. Since `backfill` is `true`, it will automatically send events for the documents that already exist in those tables during initialization.
+
+### Format of the events:
+
+The events are encoded with the "json_lines" codec, which puts compressed json documents one event per line
+
+Fields:
+
+- **db**: the database that emitted the event
+- **table**: the table that emitted the event
+- **old_val**: the old value of the document (see [changefeeds](http://rethinkdb.com/docs/changefeeds/ruby/))
+- **new_val**: the new value of the document
+- **@timestamp**: timestamp added by logstash (so may not correspond to the rethinkdb server time the change was emitted)
+- **@version**: version number added by logstash (always 1)
+
+## Known limitations
+
+There are two limitations that should be known by anyone using this in production systems:
+
+1. Until RethinkDB supports [resuming changefeeds](https://github.com/rethinkdb/rethinkdb/issues/3471), this plugin cannot guarantee that no changes are missed if a connection to the database is dropped. Again, once that functionality is implemented, this plugin will be modified to provide reliable "at least once" semantics for changes (meaning once it reconnects, it can catch back up and send any changes that it missed).
+2. Documents that are deleted in RethinkDB while the LogStash plugin is disconnected will not be synchronized. This is true even if `backfill` is enabled. This limitation is a consequence of LogStash operating on a document-by-document basis.
+
+## License
+
+Apache 2.0

data/lib/logstash/inputs/rethinker.rb

@@ -0,0 +1,279 @@
+require "logstash/inputs/base"
+require "logstash/namespace"  
+require "eventmachine"
+require "rethinkdb"
+require "stud/interval"
+require "socket" 
+
+class LogStash::Inputs::Rethinker < LogStash::Inputs::Base
+  config_name "rethinker"
+
+  # If undefined, Logstash will complain, even if codec is unused.
+  default :codec, "json_lines"
+
+  include RethinkDB::Shortcuts
+
+  # Hostname of RethinkDB server
+  config :host, :validate => :string, :default => "localhost"
+  # Driver connection port of RethinkDB server
+  config :port, :validate => :number, :default => 28015
+  # Auth key of RethinkDB server (don't provide if nil)
+  config :auth_key, :validate => :string, :default => ""
+  # Time period to squash changefeeds on. Defaults to no squashing.
+  config :squash, :default => true
+  # Which tables to watch for changes
+  config :watch_tables, :validate => :array, :default => ["test.foo", "test.bar"]
+  # Which databases to watch for changes. Tables added or removed from
+  # these databases will be watched or unwatched accordingly
+  config :watch_dbs, :validate => :array, :default => ["test"]
+  # Whether to backfill documents from the dbs and tables when
+  # (re)connecting to RethinkDB. This ensures all documents in the
+  # RethinkDB tables will be sent over logstash, but it may cause a
+  # lot of traffic with very large tables and/or unstable connections.
+  config :backfill, :default => true
+  #ssl support
+  config :ca_certs, :default => nil
+  # Credentials as of RethinkDB v2.3.x
+  config :user, :validate => :string, :default => "admin"
+  config :password, :validate => :string, :default => ""
+
+  public
+  def register
+    # {db => {table => QueryHandle}}
+    @table_feeds = Hash.new { |hsh, key| hsh[key] = {} }
+    # {db => QueryHandle}
+    @db_feeds = {}
+    @queue = nil
+    @backfill = @backfill && @backfill != 'false'
+    @squash = @squash && @squash != 'false'
+  end # def register
+
+  def run(queue)
+    @queue = queue
+    if @ca_certs
+      ssl = { :ca_certs => @ca_certs }
+    else
+      ssl = nil
+    end
+
+    unless @auth_key == ""
+      @conn = r.connect(
+        :host => @host,
+        :port => @port,
+        :auth_key => @auth_key,
+        :ssl => ssl
+      )
+    else
+      @conn = r.connect(
+        :host => @host,
+        :port => @port,
+        :user => @user,
+        :password => @password,
+        :ssl => ssl
+      )
+    end
+
+    EM.run do
+      @logger.info "Eventmachine loop started"
+      @watch_dbs.uniq.each do |db|
+        create_db_feed(db, DBHandler.new(db, self))
+      end
+      @watch_tables.uniq.each do |db_table|
+        db, table = db_table.split '.'
+        db, table = "test", db if table.nil?
+        update_db_tables(nil, {'db' => db, 'name' => table})
+      end
+    end
+  end # def run
+
+  def send(db, table, old_val, new_val)
+    event = LogStash::Event.new(
+      "db" => db,
+      "table" => table,
+      "old_val" => old_val,
+      "new_val" => new_val
+    )
+    decorate(event)
+    @queue << event
+  end
+
+  def update_db_tables(old_val, new_val)
+    unless new_val.nil?
+      handler = TableHandler.new(new_val['db'], new_val['name'], self)
+      create_table_feed(new_val['db'], new_val['name'], handler)
+    end
+    unless old_val.nil?
+      unregister_table(old_val['db'], old_val['name'], nil)
+    end
+  end
+
+  def register_table(db, table, qhandle)
+    # Add a table feed to the registry
+    unless @table_feeds.has_key?(db) &&
+           @table_feeds[db].has_key?(table)
+      @logger.info("Watching table #{db}.#{table}")
+      @table_feeds[db][table] = qhandle
+    else
+      qhandle.close
+    end
+  end
+
+  def unregister_table(db, table, qhandle)
+    # Remove a table from the registry.
+    if @table_feeds.has_key?(db) &&
+       @table_feeds[db].has_key?(table) &&
+       # If a duplicate feed comes in for the same table and needs to
+       # be unregistered, we need to check if the handle is the same
+       (qhandle.nil? || @table_feeds[db][table].equal?(qhandle))
+      @logger.info("Unregistering table #{db}.#{table}")
+      @table_feeds[db].delete(table).close
+    end
+  end
+
+  def register_db(db, qhandle)
+    # Add a db to the registry to watch it for updates to which tables
+    # are listed in it.
+    unless @db_feeds.has_key? db
+      @db_feeds[db] = qhandle
+      @logger.info "Feed for db '#{db}' registered"
+    end
+  end
+
+  def unregister_db(db)
+    # Remove a db from the registry, close all of its feeds, and
+    # remove its entry in @db_admin_tables
+    if @table_feeds.has_key?(db)
+      @logger.info("Unregistering feed for db '#{db}'")
+      @table_feeds[db].keys.each do |table|
+        unregister_table(db, table, nil)
+      end
+      @table_feeds.delete(db)
+      @db_feeds.delete(db).close
+    end
+  end
+
+  def create_db_feed(db, handler)
+    r.db('rethinkdb').
+      table('table_status').
+      changes(:include_initial => @backfill,
+              :squash => @squash,
+              :include_states => true).
+      # The filter and pluck are after .changes due to bug #5241. When
+      # that's solved they can be moved before .changes and can be
+      # simplified since they won't have to operate over both new_val
+      # and old_val
+      filter{|row| r([row['old_val']['db'], row['new_val']['db']]).contains(db).
+              and(row['status']['all_replicas_ready'])}.
+      pluck({:new_val => ['db', 'name'], :old_val => ['db', 'name']}).
+      em_run(@conn, handler)
+  end
+
+  def create_table_feed(db, table, handler)
+    options = {
+      :time_format => 'raw',
+      :binary_format => 'raw',
+    }
+    r.db(db).
+      table(table).
+      changes(:include_initial => @backfill,
+              :squash => @squash,
+              :include_states => true).
+      em_run(@conn, options, handler)
+  end
+
+  def teardown
+    # Goes through all existing handles and closes them, then clears
+    # out the registry and closes the connection to RethinkDB
+    @table_feeds.values.each do |tables|
+      tables.values.each { |qhandle| qhandle.close }
+    end
+    @db_feeds.values.each { |qhandle| qhandle.close }
+    @table_feeds.clear
+    @db_feeds.clear
+    @conn.close
+    EM.stop
+    @queue = nil
+  end
+
+  def stop
+    # nothing to do in this case so it is not necessary to define stop
+    # examples of common "stop" tasks:
+    #  * close sockets (unblocking blocking reads/accepts)
+    #  * cleanup temporary files
+    #  * terminate spawned threads
+  end
+end # class LogStash::Inputs::Rethinker
+
+# This handles feeds listening for changes to documents in a table
+class TableHandler < RethinkDB::Handler
+  attr_accessor :db
+  attr_accessor :table
+  def initialize(db, table, plugin)
+    super()
+    @db = db
+    @table = table
+    @plugin = plugin
+  end
+
+  def on_initial_val(val)
+    @plugin.send(@db, @table, nil, val)
+  end
+
+  def on_change(old_val, new_val)
+    @plugin.send(@db, @table, old_val, new_val)
+  end
+
+  def on_open(qhandle)
+    @plugin.register_table(@db, @table, qhandle)
+  end
+
+  def on_close(qhandle)
+    @plugin.unregister_table(@db, @table, qhandle)
+  end
+
+  def on_error(err, qhandle)
+    @plugin.logger.error(err.to_s)
+    @plugin.unregister_table(@db, @table, qhandle)
+  end
+
+  def on_change_error(err_str)
+    @plugin.logger.warn(err.to_s)
+  end
+end
+
+# Handler for changes to the tables in a database
+class DBHandler < RethinkDB::Handler
+
+  attr_accessor :db
+
+  def initialize(db, plugin)
+    super()
+    @db = db
+    @plugin = plugin
+  end
+
+  def on_open(qhandle)
+    @plugin.register_db(@db, qhandle)
+  end
+
+  def on_close(qhandle)
+    @plugin.unregister_db(@db)
+  end
+
+  def on_error(err, qhandle)
+    @plugin.logger.error(err.to_s)
+    @plugin.unregister_db(@db)
+  end
+
+  def on_change_error(err_str)
+    @plugin.logger.warn(err_str)
+  end
+
+  def on_initial_val(val)
+    @plugin.update_db_tables(nil, val)
+  end
+
+  def on_change(old_val, new_val)
+    @plugin.update_db_tables(old_val, new_val)
+  end
+end

data/logstash-input-rethinker.gemspec

@@ -0,0 +1,27 @@
+Gem::Specification.new do |s|
+  s.name          = 'logstash-input-rethinker'
+  s.version       = '0.1.0'
+  s.licenses      = ['Apache License (2.0)']
+  s.summary       = 'RethinkDB updated logstash plugin'
+  s.description   = 'Copy paste of old RethinkDB plugin to new logstash format'
+  s.homepage      = 'https://github.com/wayann/logstash-input-rethinker'
+  s.authors       = ['Wayann']
+  s.email         = 'wayann@nrg.re'
+  s.require_paths = ['lib']
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
+
+  # Gem dependencies
+  s.add_runtime_dependency 'logstash-core-plugin-api', '~> 2.0'
+  s.add_runtime_dependency 'logstash-codec-plain', '~> 3.0', '>= 3.0.2'
+  s.add_runtime_dependency 'stud', '~> 0.0.22'
+  s.add_runtime_dependency 'rethinkdb', '~> 2.0', '>= 2.3.0'
+  s.add_runtime_dependency 'eventmachine', '~> 1.0', '>= 1.0.9.1'
+  s.add_development_dependency 'logstash-devutils', '~> 1.3', '>= 1.3.1'
+end

data/spec/inputs/rethinker_spec.rb

@@ -0,0 +1,11 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"
+# require "logstash/inputs/rethinker"
+
+# describe LogStash::Inputs::Rethinker do
+
+#   it_behaves_like "an interruptible input plugin" do
+#     let(:config) { { "interval" => 100 } }
+#   end
+
+# end

metadata

@@ -0,0 +1,163 @@
+--- !ruby/object:Gem::Specification
+name: logstash-input-rethinker
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Wayann
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2017-03-24 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  name: logstash-core-plugin-api
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 3.0.2
+  name: logstash-codec-plain
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 3.0.2
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.0.22
+  name: stud
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.0.22
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.3.0
+  name: rethinkdb
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.3.0
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.0.9.1
+  name: eventmachine
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.0.9.1
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.3.1
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.3.1
+description: Copy paste of old RethinkDB plugin to new logstash format
+email: wayann@nrg.re
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- DEVELOPER.md
+- Gemfile
+- LICENSE
+- README.md
+- lib/logstash/inputs/rethinker.rb
+- logstash-input-rethinker.gemspec
+- spec/inputs/rethinker_spec.rb
+homepage: https://github.com/wayann/logstash-input-rethinker
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: input
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.6.8
+signing_key:
+specification_version: 4
+summary: RethinkDB updated logstash plugin
+test_files:
+- spec/inputs/rethinker_spec.rb