logstash-output-charrington 0.2.0 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d8166adfa6e084d6e0fd56921edbc8b71dae22e6e5e282c86f73753692f14dbf
-  data.tar.gz: 5b0286ebc0540a5152a11ffacd441d465c6cc44b7c67e66a4b08f5c825588dca
+  metadata.gz: fbbae45d9c763ffa27448320f7150f9827732ef5b20823c078664d6deb8ab652
+  data.tar.gz: c18245e0ac6cbb11e30a824fdaf2a39dcf466e4631f307a9b1bd6aa9484bb5e1
 SHA512:
-  metadata.gz: b977663c714b04890525419c98773e80f2e55993ff3cfc0e50ca3dee6d2b06e81a948a39d1e0004ba156ed219482592ddb0f604514310a95fd6e9ac66312db5d
-  data.tar.gz: 32a8eee42bf6a88c3c3a244805546edc2a5cbe67626bd5b12450c2c963b1967f7aea2821cfbd068478a71a2364f4ca24e1fc5759c76f7ebb6bdffb64c5011e43
+  metadata.gz: d139e03b049aae067a3e5827c520eaa855e33fb889e33f51cec8cf308bb0923f04cca2b01b1b561c2578ccbe1c2ec731e6cf59918cbb101df055058403ef99c7
+  data.tar.gz: 898bb2a57d84e835a3b2e4933ebb387d1f0e030f6dc06f5c98c284aea4c51b5b6401401d85fb6bb842988d7c15497331d960735a7e57e5d9dd43c79d3f30a284

data/README.md

@@ -1,94 +1,55 @@
-# logstash-output-jdbc 
-
-[![Build Status](https://travis-ci.org/theangryangel/logstash-output-jdbc.svg?branch=master)](https://travis-ci.org/theangryangel/logstash-output-jdbc) [![Flattr this git repo](http://api.flattr.com/button/flattr-badge-large.png)](https://flattr.com/submit/auto?user_id=the_angry_angel&url=https://github.com/the_angry_angel/logstash-output-jdbc&title=logstash-output-jdbc&language=&tags=github&category=software)
-
-This plugin is provided as an external plugin and is not part of the Logstash project.
-
-This plugin allows you to output to SQL databases, using JDBC adapters.
-See below for tested adapters, and example configurations.
-
-This has not yet been extensively tested with all JDBC drivers and may not yet work for you.
-
-If you do find this works for a JDBC driver without an example, let me know and provide a small example configuration if you can.
-
-This plugin does not bundle any JDBC jar files, and does expect them to be in a
-particular location. Please ensure you read the 4 installation lines below.
-
-## Support & release schedule
-I no longer have time at work to maintain this plugin in step with Logstash's releases, and I am not completely immersed in the Logstash ecosystem. If something is broken for you I will do my best to help, but I cannot guarantee timeframes.
-
-Pull requests are always welcome. 
-
-## Changelog
-See CHANGELOG.md
-
-## Versions
-Released versions are available via rubygems, and typically tagged.
-
-For development:
-  - See master branch for logstash v5 & v6 :warning: This is untested under Logstash 6.3 at this time, and there has been 1 unverified report of an issue. Please use at your own risk until I can find the time to evaluate and test 6.3.
-  - See v2.x branch for logstash v2
-  - See v1.5 branch for logstash v1.5 
-  - See v1.4 branch for logstash 1.4
-
-## Installation
-  - Run `bin/logstash-plugin install logstash-output-charrington` in your logstash installation directory
-  - Now either:
-    - Use driver_jar_path in your configuraton to specify a path to your jar file
-  - Or:
-    - Create the directory vendor/jar/jdbc in your logstash installation (`mkdir -p vendor/jar/jdbc/`)
-    - Add JDBC jar files to vendor/jar/jdbc in your logstash installation
-  - And then configure (examples can be found in the examples directory)
-
-## Configuration options
-
-| Option                       | Type             | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | Required? | Default |
-| ------                       | ----             | -----------                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | --------- | ------- |
-| driver_class                 | String           | Specify a driver class if autoloading fails                                                                                                                                                                                                                                                                                                                                                                                                                                                   | No        |         |
-| driver_auto_commit           | Boolean          | If the driver does not support auto commit, you should set this to false                                                                                                                                                                                                                                                                                                                                                                                                                      | No        | True    |
-| driver_jar_path              | String           | File path to jar file containing your JDBC driver. This is optional, and all JDBC jars may be placed in $LOGSTASH_HOME/vendor/jar/jdbc instead.                                                                                                                                                                                                                                                                                                                                               | No        |         |
-| connection_string            | String           | JDBC connection URL                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Yes       |         |
-| connection_test              | Boolean          | Run a JDBC connection test. Some drivers do not function correctly, and you may need to disable the connection test to supress an error. Cockroach with the postgres JDBC driver is such an example.                                                                                                                                                                                                                                                                                          | No        | Yes     |
-| connection_test_query        | String           | Connection test and init query string, required for some JDBC drivers that don't support isValid(). Typically you'd set to this "SELECT 1"                                                                                                                                                                                                                                                                                                                                                    | No        |         |
-| username                     | String           | JDBC username - this is optional as it may be included in the connection string, for many drivers                                                                                                                                                                                                                                                                                                                                                                                             | No        |         |
-| password                     | String           | JDBC password - this is optional as it may be included in the connection string, for many drivers                                                                                                                                                                                                                                                                                                                                                                                             | No        |         |
-| statement                    | Array            | An array of strings representing the SQL statement to run. Index 0 is the SQL statement that is prepared, all other array entries are passed in as parameters (in order). A parameter may either be a property of the event (i.e. "@timestamp", or "host") or a formatted string (i.e. "%{host} - %{message}" or "%{message}"). If a key is passed then it will be automatically converted as required for insertion into SQL. If it's a formatted string then it will be passed in verbatim. | Yes       |         |
-| unsafe_statement             | Boolean          | If yes, the statement is evaluated for event fields - this allows you to use dynamic table names, etc. **This is highly dangerous** and you should **not** use this unless you are 100% sure that the field(s) you are passing in are 100% safe. Failure to do so will result in possible SQL injections. Example statement: [ "insert into %{table_name_field} (column) values(?)", "fieldname" ]                                                                                            | No        | False   |
-| max_pool_size                | Number           | Maximum number of connections to open to the SQL server at any 1 time                                                                                                                                                                                                                                                                                                                                                                                                                         | No        | 5       |
-| connection_timeout           | Number           | Number of milliseconds before a SQL connection is closed                                                                                                                                                                                                                                                                                                                                                                                                                                           | No        | 10000   |
-| flush_size                   | Number           | Maximum number of entries to buffer before sending to SQL - if this is reached before idle_flush_time                                                                                                                                                                                                                                                                                                                                                                                         | No        | 1000    |
-| max_flush_exceptions         | Number           | Number of sequential flushes which cause an exception, before the set of events are discarded. Set to a value less than 1 if you never want it to stop. This should be carefully configured with respect to retry_initial_interval and retry_max_interval, if your SQL server is not highly available                                                                                                                                                                                         | No        | 10      |
-| retry_initial_interval       | Number           | Number of seconds before the initial retry in the event of a failure. On each failure it will be doubled until it reaches retry_max_interval                                                                                                                                                                                                                                                                                                                                                  | No        | 2       |
-| retry_max_interval           | Number           | Maximum number of seconds between each retry                                                                                                                                                                                                                                                                                                                                                                                                                                                  | No        | 128     |
-| retry_sql_states             | Array of strings | An array of custom SQL state codes you wish to retry until `max_flush_exceptions`. Useful if you're using a JDBC driver which returns retry-able, but non-standard SQL state codes in it's exceptions.                                                                                                                                                                                                                                                                                        | No        | []      |
-| event_as_json_keyword        | String           | The magic key word that the plugin looks for to convert the entire event into a JSON object. As Logstash does not support this out of the box with it's `sprintf` implementation, you can use whatever this field is set to in the statement parameters                                                                                                                                                                                                                                       | No        | @event  |
-| enable_event_as_json_keyword | Boolean          | Enables the magic keyword set in the configuration option `event_as_json_keyword`. Without this enabled the plugin will not convert the `event_as_json_keyword` into JSON encoding of the entire event.                                                                                                                                                                                                                                                                                       | No        | False   |
-| schema                       | String           | Database schema                                                                                                                                                                                                                                                                                      | No        | False   |
-
-## Example configurations
-Example logstash configurations, can now be found in the examples directory. Where possible we try to link every configuration with a tested jar.
-
-If you have a working sample configuration, for a DB thats not listed, pull requests are welcome.
-
-## Development and Running tests
-For development tests are recommended to run inside a virtual machine (Vagrantfile is included in the repo), as it requires
-access to various database engines and could completely destroy any data in a live system.
-
-If you have vagrant available (this is temporary whilst I'm hacking on v5 support. I'll make this more streamlined later):
-  - `vagrant up`
-  - `vagrant ssh`
-  - `cd /vagrant`
-  - `gem install bundler`
-  - `cd /vagrant && bundle install && bundle exec rake vendor && bundle exec rake install_jars`
-  - `./scripts/travis-before_script.sh && source ./scripts/travis-variables.sh`
-  - `bundle exec rspec`
-
-## Releasing
-  - Update Changelog
-  - Bump version in gemspec
-  - Commit
-  - Create tag `git tag v<version-number-in-gemspec>`
-  - `bundle exec rake install_jars`
-  - `bundle exec rake pre_release_checks`
-  - `gem build logstash-output-jdbc.gemspec`
-  - `gem push`
+# Logstash Output Charrington
+
+Logstash Output Plugin to handle batching messages and dynamically create and alter tables.
+
+Forked From [logstash-output-jdbc](https://github.com/theangryangel/logstash-output-jdbc)
+## Configure
+Example Logstash Config
+```bash
+input {
+  kafka {
+    auto_offset_reset => earliest
+    bootstrap_servers => "${LS_KAFKA_BOOTSTRAP_SERVERS:kafka:9092}"
+    codec => json
+    group_id => "winston_ls_pg_sinks_cg"
+    topics => [
+      "orwell.analytics.v1.json"
+    ]
+  }
+}
+
+filter {
+  if "_jsonparsefailure" in [tags] {
+    drop { }
+  }
+
+  ruby {
+    code => "
+      hash = event.to_hash
+      hash.each do |k,v|
+        if v == nil
+          event.remove(k)
+        end
+      end
+    "
+  }
+}
+
+output {
+  charrington {
+    driver_jar_path => "${DRIVER_PATH:/usr/share/logstash/vendor/jar/jdbc/postgresql-42.2.5.jar}"
+    connection_string => "jdbc:postgresql://${WINSTON_DATABASE_HOST}:5432/${WINSTON_DATABASE}?user=${WINSTON_DATABASE_USERNAME}&password=${WINSTON_DATABASE_PASSWORD}"
+    schema => "dea"
+  }
+
+  if "${LS_SHOW_DEBUG_OUTPUT}" == "true" {
+    stdout { codec => rubydebug }
+  }
+}
+```
+
+## Build & Publish
+```bash
+gem build logstash-output-charrington.gemspec       # build
+gem push logstash-output-charrington-x.x.x.gem      # publish
+```

data/lib/logstash/outputs/charrington.rb

@@ -23,27 +23,10 @@ require File.join(File.dirname(__FILE__), "charrington/insert")
 
 class LogStash::Outputs::Charrington < LogStash::Outputs::Base
   concurrency :shared
+  config_name 'charrington'
 
   STRFTIME_FMT = '%Y-%m-%d %T.%L'.freeze
 
-  # RETRYABLE_SQLSTATE_CLASSES = [
-  #   # Classes of retryable SQLSTATE codes
-  #   # Not all in the class will be retryable. However, this is the best that
-  #   # we've got right now.
-  #   # If a custom state code is required, set it in retry_sql_states.
-  #   '08', # Connection Exception
-  #   '24', # Invalid Cursor State (Maybe retry-able in some circumstances)
-  #   '25', # Invalid Transaction State
-  #   '40', # Transaction Rollback
-  #   '53', # Insufficient Resources
-  #   '54', # Program Limit Exceeded (MAYBE)
-  #   '55', # Object Not In Prerequisite State
-  #   '57', # Operator Intervention
-  #   '58', # System Error
-  # ].freeze
-
-  config_name 'charrington'
-
   # Driver class - Reintroduced for https://github.com/theangryangel/logstash-output-jdbc/issues/26
   config :driver_class, validate: :string
 
@@ -120,8 +103,11 @@ class LogStash::Outputs::Charrington < LogStash::Outputs::Base
                schema: @schema,
                max_retries: @max_flush_exceptions,
                retry_initial_interval: @retry_initial_interval }
-
       Charrington::Process.call(connection, event, opts)
+    rescue => e
+      @logger.error("Unable to process event. Event dropped. #{e.message}")
+      next
+    ensure
       connection.close unless connection.nil?
     end
   end
@@ -201,8 +187,6 @@ class LogStash::Outputs::Charrington < LogStash::Outputs::Base
     log_method = (retrying ? 'warn' : 'error')
 
     loop do
-      # TODO reformat event output so that it only shows the fields necessary.
-
       @logger.send(log_method, log_text, :exception => current_exception, :event => event)
 
       if current_exception.respond_to? 'getNextException'

data/lib/logstash/outputs/charrington/alter_table.rb

@@ -14,7 +14,7 @@ module Charrington
 
     def initialize(connection, event, table_name, columns)
       @connection = connection
-      @event = event
+      @event = event.to_hash
       @table_name = table_name
       @columns = columns
       @column_types = []
@@ -77,7 +77,7 @@ module Charrington
       stmt = connection.prepareStatement(prep_sql(sql))
       stmt.execute()
     rescue Java::OrgPostgresqlUtil::PSQLException => e
-      # @logger.error("#{e.message}")
+      puts "PSQLException: #{e.message}"
     ensure
       stmt.close unless stmt.nil?
     end

data/lib/logstash/outputs/charrington/create_table.rb

@@ -17,7 +17,7 @@ module Charrington
       @event = event.to_hash
       @table_name = table_name
       @columns = columns
-      @column_types = []
+      @column_types = initial_columns
     end
 
     def call
@@ -53,6 +53,11 @@ module Charrington
       end
     end
 
+    def initial_columns
+      # Postgres
+      ["id SERIAL PRIMARY KEY", "inserted_at TIMESTAMP DEFAULT NOW()"]
+    end
+
     def create_table
       execute("CREATE TABLE IF NOT EXISTS #{table_name} (#{column_types.join(', ')})")
     end
@@ -61,7 +66,7 @@ module Charrington
       statement = connection.prepareStatement( sql.gsub(/\s+/, " ").strip )
       statement.execute()
     rescue Java::OrgPostgresqlUtil::PSQLException => e
-      # @logger.error("#{e.message}")
+      puts "PSQLException: #{e.message}"
     ensure
       statement.close unless statement.nil?
     end

data/lib/logstash/outputs/charrington/insert.rb

@@ -133,7 +133,7 @@ module Charrington
       statement = connection.prepareStatement( sql.gsub(/\s+/, " ").strip )
       statement.execute()
     rescue Java::OrgPostgresqlUtil::PSQLException => e
-      @logger.error("#{e.message}")
+      puts "PSQLException: #{e.message}"
     ensure
       statement.close unless statement.nil?
     end

data/lib/logstash/outputs/charrington/transform.rb

@@ -9,14 +9,17 @@ module Charrington
     Error = Class.new(StandardError)
     EventNil = Class.new(Error)
     TableNameNil = Class.new(Error)
+    ColumnBlacklist = Class.new(Error)
 
-    KEY_BLACKLIST = ['host','path','jwt']
+    KEY_FILTER_BLACKLIST = ['host','path','jwt','sequence']
+    KEY_RAISE_BLACKLIST = ['id', 'inserted_at']
 
     def initialize(event)
       raise EventNil, "Event is nil" if event.nil?
-      event = event.is_a?(Hash) ? event : event.to_hash
+      event = event.to_hash
       @event = drop_keys(event)
       @top_level_keys = @event.keys
+      check_blacklist
     end
 
     def call
@@ -28,8 +31,14 @@ module Charrington
 
     private
 
+    def check_blacklist
+      arr = []
+      KEY_RAISE_BLACKLIST.each { |k| arr << k if event.keys.include?(k) }
+      raise ColumnBlacklist, "Event contains these blacklisted keys: #{arr.join(",")}" unless arr.empty?
+    end
+
     def drop_keys(event)
-      event.delete_if {|k, _v| k.start_with?("@") || KEY_BLACKLIST.include?(k) }
+      event.delete_if {|k, _v| k.start_with?("@") || KEY_FILTER_BLACKLIST.include?(k) }
     end
 
     def flatten_hash(hash)

data/logstash-output-charrington.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name = 'logstash-output-charrington'
-  s.version = '0.2.0'
+  s.version = '0.2.1'
 
   s.licenses = ['Apache License (2.0)']
   s.summary = 'This plugin allows you to output to SQL, via JDBC'
@@ -30,4 +30,5 @@ Gem::Specification.new do |s|
   s.add_development_dependency 'rubocop', '0.41.2'
   s.add_development_dependency 'logstash-input-generator'
   s.add_development_dependency 'logstash-codec-json'
+  s.add_development_dependency 'insist'
 end

data/spec/outputs/charrington_spec.rb

@@ -1,4 +1,5 @@
 require_relative '../charrington_spec_helper'
+require 'insist'
 
 describe LogStash::Outputs::Charrington do
   describe 'when initializing' do
@@ -9,36 +10,55 @@ describe LogStash::Outputs::Charrington do
     end
   end
 
+  # describe 'integration tests with agent' do
+  #   config <<-HEREDOC
+  #             input {
+  #               generator {
+  #                 message => '{"app_name": "Web App", "event": "From Agent"}'
+  #                 codec => 'json'
+  #                 count => 1
+  #               }
+  #             }
+
+  #             output {
+  #               charrington {
+  #                 connection_string => 'jdbc:postgresql://localhost:5432/winston?user=postgres&password=postgres'
+  #                 driver_jar_path => '/projects/logstash-output-charrington/vendor/postgresql-42.2.5.jar'
+  #                 schema => 'dea'
+  #               }
+  #             }
+  #             HEREDOC
+
+  #   agent do
+  #     puts "IT'S WORKING!!!!!"
+  #   end
+  # end
+
   describe 'integration tests' do
-    config <<-CONFIG
-    input {
-      generator {
-        message => '{"id": "abc", "app_name": "Web App", "event": "Hi - Dan"}'
-        codec => 'json'
-        count => 1
+    let(:config) do <<-CONFIG
+      input {
+        generator {
+          message => '{"app_name": "Web App", "event": "Hi - Dan"}'
+          codec => 'json'
+          count => 1
+        }
       }
-    }
 
-    output {
-      charrington {
-        connection_string => 'jdbc:postgresql://localhost:5432/winston?user=postgres&password=postgres'
-        driver_jar_path => '/projects/logstash-output-charrington/vendor/postgresql-42.2.5.jar'
-        schema => 'dea'
+      output {
+        charrington {
+          connection_string => 'jdbc:postgresql://localhost:5432/winston?user=postgres&password=postgres'
+          driver_jar_path => '/projects/logstash-output-charrington/vendor/postgresql-42.2.5.jar'
+          schema => 'dea'
+        }
       }
-    }
-    CONFIG
+      CONFIG
+    end
 
-    agent do
-      puts "IT'S WORKING!!!!!"
+    it("isn't effed") do
+      pipeline = new_pipeline_from_string(config)
+      pipeline.run
     end
   end
-
-  # context 'running' do
-  #   it 'should transform some JSON' do
-  #     transformed = Charrington::Transform.call({"a" => 1})
-  #     puts transformed
-  #   end
-  # end
 end
 
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: logstash-output-charrington
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.2.1
 platform: ruby
 authors:
 - dconger
@@ -10,7 +10,7 @@ authors:
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2019-06-17 00:00:00.000000000 Z
+date: 2019-06-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -130,6 +130,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: insist
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: This gem is a logstash plugin required to be installed on top of the
   Logstash core pipeline using $LS_HOME/bin/logstash-plugin install 'logstash-output-charrington'.
   This gem is not a stand-alone program
@@ -144,6 +158,8 @@ files:
 - README.md
 - THANKS.md
 - lib/com/zaxxer/HikariCP/2.7.2/HikariCP-2.7.2.jar
+- lib/commons-io/commons-io/2.4/commons-io-2.4.jar
+- lib/de/flapdoodle/embed/de.flapdoodle.embed.process/2.0.2/de.flapdoodle.embed.process-2.0.2.jar
 - lib/logstash-output-charrington_jars.rb
 - lib/logstash/outputs/charrington.rb
 - lib/logstash/outputs/charrington/alter_table.rb
@@ -152,9 +168,16 @@ files:
 - lib/logstash/outputs/charrington/process.rb
 - lib/logstash/outputs/charrington/service.rb
 - lib/logstash/outputs/charrington/transform.rb
+- lib/net/java/dev/jna/jna-platform/4.0.0/jna-platform-4.0.0.jar
+- lib/net/java/dev/jna/jna/4.0.0/jna-4.0.0.jar
+- lib/org/apache/commons/commons-compress/1.10/commons-compress-1.10.jar
+- lib/org/apache/commons/commons-lang3/3.1/commons-lang3-3.1.jar
 - lib/org/apache/logging/log4j/log4j-api/2.6.2/log4j-api-2.6.2.jar
 - lib/org/apache/logging/log4j/log4j-slf4j-impl/2.6.2/log4j-slf4j-impl-2.6.2.jar
+- lib/org/postgresql/postgresql/42.2.5/postgresql-42.2.5.jar
 - lib/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar
+- lib/ru/yandex/qatools/embed/postgresql-embedded/2.10/postgresql-embedded-2.10.jar
+- lib/ru/yandex/qatools/embed/postgresql-embedded/2.8/postgresql-embedded-2.8.jar
 - logstash-output-charrington.gemspec
 - spec/charrington_spec_helper.rb
 - spec/outputs/charrington_mysql_spec.rb