logstash-codec-avro_schema_registry 0.9.1 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e3deca295d01556563ba3b141432f7c99e6bfd86
-  data.tar.gz: 15b37b40e9b9a347b5a88bd794aec5fc390d5320
+  metadata.gz: 4b52cb641d3dcc1ab0e81aac04e073a609cf55e4
+  data.tar.gz: 26d12ac1f854fb677f7f8ecb23da35967e88a3b5
 SHA512:
-  metadata.gz: ae5dd705ee5835867fa040b37fed57d342bc6a58e5845b07b98349c3b524cd168863c83683fded460c366b3d12beb4c796af8ee7d0c147b84fc50417722e1bb0
-  data.tar.gz: 21828534180e3dc2527fd794fb6b3622b2f724d4cbfdc8dcbfc4e2f1b0eab682fb57315c24264bf617e65fe42ea7632b7dd7bdb70dc75191779e17f0180b2c42
+  metadata.gz: 3307ec8a0891959b5f1e9f6860463de2b108b70f9f0a9103106bfc58e8e81b8fc7fd0a463524c88f19e684e28cf209dc9a1f3c833218be252beb8c4dc7ae3a9a
+  data.tar.gz: d49ca291bc9b1e4ff75d3bbcd9feec6657d22cdd596e4523d409acda8c6b0d5af7d44736aa13f5f8c08fbfc539ea9b34c5a5e2f1df8dd33014f19f03856283ad

data/CHANGELOG.md

@@ -1,3 +1,7 @@
+## 1.0.0
+ - Add encode capability
+ - Try base64 decode/encode on input/output
+
 ## 0.9.1
  - Upgrade to logstash 5.1
 

data/DEVELOPER.md

@@ -5,7 +5,7 @@ How to Install
 --------------
 
 ```
-bin/plugin install logstash-avro_schema_registry-codec
+bin/plugin install logstash-codec-avro_schema_registry
 ```
 
 How to Use

data/README.md

@@ -1,86 +1,84 @@
-# Logstash Plugin
+# Logstash Codec - Avro Schema Registry
 
-This is a plugin for [Logstash](https://github.com/elastic/logstash).
+This plugin is used to serialize Logstash events as
+Avro datums, as well as deserializing Avro datums into
+Logstash events.
 
-It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+Decode/encode Avro records as Logstash events using the 
+associated Avro schema from a Confluent schema registry.
+(https://github.com/confluentinc/schema-registry)
 
-## Documentation
+##  Decoding (input)
 
-Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+When this codec is used to decode the input, you may pass the following options:
+- ``endpoint`` - always required.
+- ``username`` - optional.
+- ``password`` - optional.
 
-- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
-- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+If the input stream is binary encoded, you should use the ``ByteArrayDeserializer``
+in the Kafka input config.
 
-## Need Help?
+## Encoding (output)
 
-Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+This codec uses the Confluent schema registry to register a schema and
+encode the data in Avro using schema_id lookups.
 
-## Developing
+When this codec is used to encode, you may pass the following options:
+- ``endpoint`` - always required.
+- ``username`` - optional.
+- ``password`` - optional.
+- ``schema_id`` - when provided, no other options are required.
+- ``subject_name`` - required when there is no ``schema_id``.
+- ``schema_version`` - when provided, the schema will be looked up in the registry.
+- ``schema_uri`` - when provided, JSON schema is loaded from URL or file.
+- ``schema_string`` - required when there is no ``schema_id``, ``schema_version`` or ``schema_uri``
+- ``check_compatibility`` - will check schema compatibility before encoding.
+- ``register_schema`` - will register the JSON schema if it does not exist.
+- ``binary_encoded`` - will output the encoded event as a ByteArray.
+  Requires the ``ByteArraySerializer`` to be set in the Kafka output config.
 
-### 1. Plugin Developement and Testing
+## Usage
 
-#### Code
-- To get started, you'll need JRuby with the Bundler gem installed.
+### Basic usage with Kafka input and output.
 
-- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
-
-- Install dependencies
-```sh
-bundle install
-```
-
-#### Test
-
-- Update your dependencies
-
-```sh
-bundle install
-```
-
-- Run tests
-
-```sh
-bundle exec rspec
-```
-
-### 2. Running your unpublished Plugin in Logstash
-
-#### 2.1 Run in a local Logstash clone
-
-- Edit Logstash `Gemfile` and add the local plugin path, for example:
-```ruby
-gem "logstash-codec-awesome", :path => "/your/local/logstash-codec-awesome"
-```
-- Install plugin
-```sh
-bin/logstash-plugin install --no-verify
 ```
-- Run Logstash with your plugin
-```sh
-bin/logstash -e 'codec {awesome {}}'
+input {
+  kafka {
+    ...
+    codec => avro_schema_registry {
+      endpoint => "http://schemas.example.com"
+    }
+    value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
+  }
+}
+filter {
+  ...
+}
+output {
+  kafka {
+    ...
+    codec => avro_schema_registry {
+      endpoint => "http://schemas.example.com"
+      subject_name => "my_kafka_subject_name"
+      schema_uri => "/app/my_kafka_subject.avsc"
+      register_schema => true
+    }
+  }
+}
 ```
-At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
 
-#### 2.2 Run in an installed Logstash
+### Binary encoded Kafka output
 
-You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
-
-- Build your plugin gem
-```sh
-gem build logstash-codec-awesome.gemspec
-```
-- Install the plugin from the Logstash home
-```sh
-bin/logstash-plugin install /your/local/plugin/logstash-codec-awesome.gem
 ```
-- Start Logstash and proceed to test the plugin
-
-## Contributing
-
-All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
-
-Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
-
-It is more important to the community that you are able to contribute.
-
-For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
+output {
+  kafka {
+    ...
+    codec => avro_schema_registry {
+      endpoint => "http://schemas.example.com"
+      schema_id => 47
+      binary_encoded => true
+    }
+    value_serializer => "org.apache.kafka.common.serialization.ByteArraySerializer"
+  }
+}
+```

data/lib/logstash/codecs/avro_schema_registry.rb

@@ -1,5 +1,6 @@
 # encoding: utf-8
 require "avro"
+require "open-uri"
 require "schema_registry"
 require "schema_registry/client"
 require "logstash/codecs/base"
@@ -7,73 +8,185 @@ require "logstash/namespace"
 require "logstash/event"
 require "logstash/timestamp"
 require "logstash/util"
+require "base64"
 
 MAGIC_BYTE = 0
 
-# Read serialized Avro records as Logstash events
+# == Logstash Codec - Avro Schema Registry
 #
 # This plugin is used to serialize Logstash events as
 # Avro datums, as well as deserializing Avro datums into
 # Logstash events.
 #
-# ==== Encoding
+# Decode/encode Avro records as Logstash events using the
+# associated Avro schema from a Confluent schema registry.
+# (https://github.com/confluentinc/schema-registry)
 #
-# This codec currently does not encode. This might be added later.
 #
+# ==== Decoding (input)
 #
-# ==== Decoding
+# When this codec is used to decode the input, you may pass the following options:
+# - ``endpoint`` - always required.
+# - ``username`` - optional.
+# - ``password`` - optional.
 #
-# This codec is for deserializing individual Avro records. It looks up
-# the associated avro schema from a Confluent schema registry.
-# (https://github.com/confluentinc/schema-registry)
+# If the input stream is binary encoded, you should use the ``ByteArrayDeserializer``
+# in the Kafka input config.
+#
+# ==== Encoding (output)
 #
+# This codec uses the Confluent schema registry to register a schema and
+# encode the data in Avro using schema_id lookups.
+#
+# When this codec is used to encode, you may pass the following options:
+# - ``endpoint`` - always required.
+# - ``username`` - optional.
+# - ``password`` - optional.
+# - ``schema_id`` - when provided, no other options are required.
+# - ``subject_name`` - required when there is no ``schema_id``.
+# - ``schema_version`` - when provided, the schema will be looked up in the registry.
+# - ``schema_uri`` - when provided, JSON schema is loaded from URL or file.
+# - ``schema_string`` - required when there is no ``schema_id``, ``schema_version`` or ``schema_uri``
+# - ``check_compatibility`` - will check schema compatibility before encoding.
+# - ``register_schema`` - will register the JSON schema if it does not exist.
+# - ``binary_encoded`` - will output the encoded event as a ByteArray.
+#   Requires the ``ByteArraySerializer`` to be set in the Kafka output config.
 #
 # ==== Usage
-# Example usage with Kafka input.
+# Example usage with Kafka input and output.
 #
 # [source,ruby]
 # ----------------------------------
 # input {
 #   kafka {
+#     ...
 #     codec => avro_schema_registry {
 #       endpoint => "http://schemas.example.com"
 #     }
+#     value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
 #   }
 # }
 # filter {
 #   ...
 # }
 # output {
-#   ...
+#   kafka {
+#     ...
+#     codec => avro_schema_registry {
+#       endpoint => "http://schemas.example.com"
+#       subject_name => "my_kafka_subject_name"
+#       schema_uri => "/app/my_kafka_subject.avsc"
+#       register_schema => true
+#     }
+#   }
+# }
+# ----------------------------------
+#
+# Binary encoded Kafka output
+#
+# [source,ruby]
+# ----------------------------------
+# output {
+#   kafka {
+#     ...
+#     codec => avro_schema_registry {
+#       endpoint => "http://schemas.example.com"
+#       schema_id => 47
+#       binary_encoded => true
+#     }
+#     value_serializer => "org.apache.kafka.common.serialization.ByteArraySerializer"
+#   }
 # }
 # ----------------------------------
+
 class LogStash::Codecs::AvroSchemaRegistry < LogStash::Codecs::Base
   config_name "avro_schema_registry"
+  
+  EXCLUDE_ALWAYS = [ "@timestamp", "@version" ]
 
   # schema registry endpoint and credentials
   config :endpoint, :validate => :string, :required => true
   config :username, :validate => :string, :default => nil
   config :password, :validate => :string, :default => nil
 
+  config :schema_id, :validate => :number, :default => nil
+  config :subject_name, :validate => :string, :default => nil
+  config :schema_version, :validate => :number, :default => nil
+  config :schema_uri, :validate => :string, :default => nil
+  config :schema_string, :validate => :string, :default => nil
+  config :check_compatibility, :validate => :boolean, :default => false
+  config :register_schema, :validate => :boolean, :default => false
+  config :binary_encoded, :validate => :boolean, :default => false
+
   public
   def register
     @client = SchemaRegistry::Client.new(endpoint, username, password)
     @schemas = Hash.new
+    @write_schema_id = nil
   end
 
   def get_schema(schema_id)
-    if !@schemas.has_key?(schema_id)
+    unless @schemas.has_key?(schema_id)
       @schemas[schema_id] = Avro::Schema.parse(@client.schema(schema_id))
     end
     @schemas[schema_id]
   end
 
+  def load_schema_json()
+    if @schema_uri
+      open(@schema_uri).read
+    elsif @schema_string
+      @schema_string
+    else
+      @logger.error('you must supply a schema_uri or schema_string in the config')
+    end
+  end
+
+  def get_write_schema_id()
+    # If schema id is passed, just use that
+    if @schema_id
+      @schema_id
+
+    else
+      # subject_name is required
+      if @subject_name == nil
+        @logger.error('requires a subject_name')
+      else
+        subject = @client.subject(@subject_name)
+
+        # If schema_version, load from subject API
+        if @schema_version != nil
+          schema = subject.version(@schema_version)
+
+        # Otherwise, load schema json and check with registry
+        else
+          schema_json = load_schema_json
+
+          # If not compatible, raise error
+          if @check_compatibility
+            unless subject.compatible?(schema_json)
+              @logger.error('the schema json is not compatible with the subject. you should fix your schema or change the compatibility level.')
+            end
+          end
+
+          if @register_schema
+            subject.register_schema(schema_json) unless subject.schema_registered?(schema_json)
+          end
+
+          schema = subject.verify_schema(schema_json)
+        end
+
+        schema.id
+      end
+    end
+  end
+
   public
   def decode(data)
     if data.length < 5
       @logger.error('message is too small to decode')
     else
-      datum = StringIO.new(data)
+      datum = StringIO.new(Base64.strict_decode64(data)) rescue StringIO.new(data)
       magic_byte, schema_id = datum.read(5).unpack("cI>")
       if magic_byte != MAGIC_BYTE
         @logger.error('message does not start with magic byte')
@@ -88,6 +201,20 @@ class LogStash::Codecs::AvroSchemaRegistry < LogStash::Codecs::Base
 
   public
   def encode(event)
-    @logger.error('Encode has not been implemented for this codec')
+    @write_schema_id ||= get_write_schema_id
+    schema = get_schema(@write_schema_id)
+    dw = Avro::IO::DatumWriter.new(schema)
+    buffer = StringIO.new
+    buffer.write(MAGIC_BYTE.chr)
+    buffer.write([@write_schema_id].pack("I>"))
+    encoder = Avro::IO::BinaryEncoder.new(buffer)
+    eh = event.to_hash
+    eh.delete_if { |key, _| EXCLUDE_ALWAYS.include? key }
+    dw.write(eh, encoder)
+    if @binary_encoded
+       @on_event.call(event, buffer.string.to_java_bytes)
+    else
+       @on_event.call(event, Base64.strict_encode64(buffer.string))
+    end
   end
 end

data/logstash-codec-avro_schema_registry.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name          = 'logstash-codec-avro_schema_registry'
-  s.version       = '0.9.1'
+  s.version       = '1.0.0'
   s.licenses      = ['Apache License (2.0)']
   s.summary         = "Encode and decode avro formatted data from a Confluent schema registry"
   s.description     = "Encode and decode avro formatted data from a Confluent schema registry"
@@ -18,9 +18,9 @@ Gem::Specification.new do |s|
   s.metadata = { "logstash_plugin" => "true", "logstash_group" => "codec" }
 
   # Gem dependencies
-  s.add_runtime_dependency 'logstash-core-plugin-api', "~> 2.0"
-  s.add_runtime_dependency 'logstash-codec-line'
+  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
+  s.add_runtime_dependency "logstash-codec-line"
   s.add_runtime_dependency "avro"  #(Apache 2.0 license)
   s.add_runtime_dependency "schema_registry"  #(MIT license)
-  s.add_development_dependency 'logstash-devutils'
+  s.add_development_dependency "logstash-devutils"
 end

data/spec/codecs/avro_schema_registry_spec.rb

@@ -1,7 +1,7 @@
 # encoding: utf-8
 require "logstash/devutils/rspec/spec_helper"
 require 'avro'
-require 'logstash/codecs/avro'
+require 'logstash/codecs/avro_schema_registry'
 require 'logstash/event'
 
 describe LogStash::Codecs::AvroSchemaRegistry do
@@ -9,8 +9,8 @@ describe LogStash::Codecs::AvroSchemaRegistry do
   let (:test_event) { LogStash::Event.new({"foo" => "hello", "bar" => 10}) }
 
   subject do
-    allow_any_instance_of(LogStash::Codecs::AvroSchemaRegistry).to \
-      receive(:open_and_read).and_return(avro_config['schema_uri'])
+    # allow_any_instance_of(LogStash::Codecs::AvroSchemaRegistry).to \
+      # receive(:open_and_read).and_return(avro_config['schema_uri'])
     next LogStash::Codecs::AvroSchemaRegistry.new(avro_config)
   end
 

metadata

@@ -1,80 +1,86 @@
 --- !ruby/object:Gem::Specification
 name: logstash-codec-avro_schema_registry
 version: !ruby/object:Gem::Version
-  version: 0.9.1
+  version: 1.0.0
 platform: ruby
 authors:
 - RevPoint Media
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2017-05-02 00:00:00.000000000 Z
+date: 2017-09-25 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: logstash-core-plugin-api
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '2.0'
-  type: :runtime
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+  name: logstash-core-plugin-api
   prerelease: false
+  type: :runtime
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
       - !ruby/object:Gem::Version
-        version: '2.0'
+        version: '2.99'
 - !ruby/object:Gem::Dependency
-  name: logstash-codec-line
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-  type: :runtime
+  name: logstash-codec-line
   prerelease: false
+  type: :runtime
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: avro
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-  type: :runtime
+  name: avro
   prerelease: false
+  type: :runtime
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: schema_registry
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-  type: :runtime
+  name: schema_registry
   prerelease: false
+  type: :runtime
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: logstash-devutils
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-  type: :development
+  name: logstash-devutils
   prerelease: false
+  type: :development
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -95,13 +101,62 @@ files:
 - lib/logstash/codecs/avro_schema_registry.rb
 - logstash-codec-avro_schema_registry.gemspec
 - spec/codecs/avro_schema_registry_spec.rb
+- vendor/cache/avro-1.8.2.gem
+- vendor/cache/chronic_duration-0.10.6.gem
+- vendor/cache/clamp-0.6.5.gem
+- vendor/cache/coderay-1.1.1.gem
+- vendor/cache/concurrent-ruby-1.0.0-java.gem
+- vendor/cache/diff-lcs-1.3.gem
+- vendor/cache/ffi-1.9.18-java.gem
+- vendor/cache/filesize-0.0.4.gem
+- vendor/cache/fivemat-1.3.5.gem
+- vendor/cache/gem_publisher-1.5.0.gem
+- vendor/cache/gems-0.8.3.gem
+- vendor/cache/i18n-0.6.9.gem
+- vendor/cache/insist-1.0.0.gem
+- vendor/cache/jar-dependencies-0.3.11.gem
+- vendor/cache/jrjackson-0.4.2-java.gem
+- vendor/cache/jrmonitor-0.4.2.gem
+- vendor/cache/jruby-openssl-0.9.16-java.gem
+- vendor/cache/kramdown-1.14.0.gem
+- vendor/cache/logstash-codec-line-3.0.3.gem
+- vendor/cache/logstash-core-5.4.0-java.gem
+- vendor/cache/logstash-core-plugin-api-2.1.24-java.gem
+- vendor/cache/logstash-devutils-1.3.3-java.gem
+- vendor/cache/method_source-0.8.2.gem
+- vendor/cache/minitar-0.5.4.gem
+- vendor/cache/multi_json-1.12.1.gem
+- vendor/cache/numerizer-0.1.1.gem
+- vendor/cache/polyglot-0.3.5.gem
+- vendor/cache/pry-0.10.4-java.gem
+- vendor/cache/puma-2.16.0-java.gem
+- vendor/cache/rack-1.6.8.gem
+- vendor/cache/rack-protection-1.5.3.gem
+- vendor/cache/rake-12.0.0.gem
+- vendor/cache/rspec-3.6.0.gem
+- vendor/cache/rspec-core-3.6.0.gem
+- vendor/cache/rspec-expectations-3.6.0.gem
+- vendor/cache/rspec-mocks-3.6.0.gem
+- vendor/cache/rspec-support-3.6.0.gem
+- vendor/cache/rspec-wait-0.0.9.gem
+- vendor/cache/ruby-maven-3.3.12.gem
+- vendor/cache/ruby-maven-libs-3.3.9.gem
+- vendor/cache/rubyzip-1.1.7.gem
+- vendor/cache/schema_registry-0.0.4.gem
+- vendor/cache/sinatra-1.4.8.gem
+- vendor/cache/slop-3.6.0.gem
+- vendor/cache/spoon-0.0.6.gem
+- vendor/cache/stud-0.0.22.gem
+- vendor/cache/thread_safe-0.3.6-java.gem
+- vendor/cache/tilt-2.0.7.gem
+- vendor/cache/treetop-1.4.15.gem
 homepage: https://github.com/revpoint/logstash-codec-avro_schema_registry
 licenses:
 - Apache License (2.0)
 metadata:
   logstash_plugin: 'true'
   logstash_group: codec
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -116,9 +171,9 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.4.5
-signing_key: 
+rubyforge_project:
+rubygems_version: 2.4.8
+signing_key:
 specification_version: 4
 summary: Encode and decode avro formatted data from a Confluent schema registry
 test_files: