logstash-output-azuresearch 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: f31f4188a841a0cfd75d86b5f4cd893b59f26abe
+  data.tar.gz: d78ab9f954ac2cb9fa1f2ea640631b81b7ba0ee1
+SHA512:
+  metadata.gz: cad4a079a66b3f01514c0d4825349415de6341e857c1e27315b8cc38333887d629c02e732a645118921ba22043e49ea25c34213d13745b52dff6a5dc5fbf0468
+  data.tar.gz: 418f7b3bad65f7b8fe9e0b1b81475a06746142044e0d7bfaa5c7a7974dad7e14c43822d7d2262dd86138aba99b8bbed5b254243228eb08522ef9d2a8b72041a9

data/CHANGELOG.md

@@ -0,0 +1,3 @@
+## 0.1.0
+
+* Inital Release

data/CONTRIBUTORS

@@ -0,0 +1,10 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Yoichi Kawasaki (yokawasa)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,94 @@
+# Azure Search output plugin for Logstash
+
+logstash-output-azuresearch is a logstash plugin to output to Azure Search. [Logstash](https://www.elastic.co/products/logstash) is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite [destinations](https://www.elastic.co/products/logstash). [Azure Search](https://docs.microsoft.com/en-us/azure/search/search-what-is-azure-search) is a managed cloud search service provided in Microsoft Azure.
+
+## Installation
+
+You can install this plugin using the Logstash "plugin" or "logstash-plugin" (for newer versions of Logstash) command:
+```
+bin/plugin install logstash-output-azuresearch
+# or
+bin/logstash-plugin install logstash-output-azuresearch  (Newer versions of Logstash)
+```
+Please see [Logstash reference](https://www.elastic.co/guide/en/logstash/current/offline-plugins.html) for more information.
+
+## Configuration
+
+```
+output {
+    azuresearch {
+        endpoint => "https://<YOUR ACCOUNT>.search.windows.net"
+        api_key => "<AZURESEARCH API KEY>"
+        search_index => "<SEARCH INDEX NAME>"
+        column_names => ['col1','col2','col3'..]  ##  ## list of column names (array)
+        key_names => ['key1','key2','key3'..] ## list of Key names (array)
+        flush_items => <FLUSH_ITEMS_NUM>
+        flush_interval_time => <FLUSH INTERVAL TIME(sec)>
+   }
+}
+```
+
+ * **endpoint (required)** - Azure Search service endpoint URI
+ * **api\_key (required)** - Azure Search API key
+ * **search\_index (required)** - Azure Search Index name to insert records
+ * **column\_names (required)** - List of column names (array) in a target Azure search index. 1st item in column_names should be primary key
+ * **key\_names (optional)** - Default:[] (empty array). List of key names (array) in in-coming record to insert. The number of keys in key_names must be equal to the one of columns in column_names. Also the order or each item in key_names must match the one of each items in column_names.
+ * **flush_items (optional)** - Default 50. Max number of items to buffer before flushing (1 - 1000).
+ * **flush_interval_time (optional)** - Default 5. Max number of seconds to wait between flushes.
+
+## Tests
+
+Here is an example configuration where Logstash's event source and destination are configured as standard input and Azure Search respectively.
+
+### Example Configuration
+```
+input {
+    stdin {
+        codec => json_lines
+    }
+}
+
+output {
+    azuresearch {
+        endpoint => "https://<YOUR ACCOUNT>.search.windows.net"
+        api_key => "<AZURESEARCH API KEY>"
+        search_index => "<SEARCH INDEX NAME>"
+        column_names => ['id','user_name','message','created_at']
+        key_names => ['postid','user','content','posttime']
+        flush_items => 100
+        flush_interval_time => 5
+    }
+}
+```
+You can find example configuration files in logstash-output-azuresearch/examples.
+
+### Run the plugin with the example configuration
+
+Now you run logstash with the the example configuration like this:
+```
+# Test your logstash configuration before actually running the logstash
+bin/logstash -f logstash-stdin-json-to-azuresearch.conf --configtest
+# run
+bin/logstash -f logstash-stdin-json-to-azuresearch.conf
+```
+
+Here is an expected output for sample input (JSON Lines):
+
+<u>JSON Lines</u>
+```
+{ "id": "a001", "user_name": "user001", "message":"msg001", "created_at":"2016-12-28T00:01:00Z" },
+{ "id": "a002", "user_name": "user002", "message":"msg002", "created_at":"2016-12-28T00:02:00Z" },
+{ "id": "a003", "user_name": "user003", "message":"msg003", "created_at":"2016-12-28T00:03:00Z" },
+```
+<u>Output (Azure Search POST message)</u>
+```
+"value": [
+    { "@search.score": 1, "id": "a001", "user_name": "user001", "message": "msg001", "created_at": "2016-12-28T00:01:00Z" },
+    { "@search.score": 1, "id": "a002", "user_name": "user002", "message": "msg002", "created_at": "2016-12-28T00:02:00Z" },
+    { "@search.score": 1, "id": "a003", "user_name": "user003", "message": "msg003", "created_at": "2016-12-28T00:03:00Z" }
+]
+```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/yokawasa/logstash-output-azuresearch.

data/VERSION

@@ -0,0 +1 @@
+0.1.0

data/lib/logstash/outputs/azuresearch.rb

@@ -0,0 +1,105 @@
+# encoding: utf-8
+require "logstash/outputs/base"
+require "logstash/namespace"
+require "stud/buffer"
+
+class LogStash::Outputs::Azuresearch < LogStash::Outputs::Base
+  include Stud::Buffer
+
+  config_name "azuresearch"
+
+  # Azure Search Endpoint URL
+  config :endpoint, :validate => :string, :required => true
+
+  # zure Search API key
+  config :api_key, :validate => :string, :required => true
+
+  # Azure Search Index name to insert records
+  config :search_index, :validate => :string, :required => true
+
+  # Column names in a target Azure search index.
+  # 1st item in column_names should be primary key
+  config :column_names, :validate => :array, :required => true
+
+  # Key names in incomming record to insert.
+  # The number of keys in key_names must be equal to the one of columns in column_names.
+  # Also the order or each item in key_names must match the one of each items in column_names.
+  config :key_names, :validate => :array, :default => []
+
+  # Max number of items to buffer before flushing (1 - 1000). Default 50.
+  config :flush_items, :validate => :number, :default => 50
+  
+  # Max number of seconds to wait between flushes. Default 5
+  config :flush_interval_time, :validate => :number, :default => 5
+
+
+  public
+  def register
+    require_relative 'azuresearch/client'
+
+    ## Configure
+    if @key_names.length < 1
+      @key_names = @column_names
+    end
+    raise ArgumentError, 'NOT match keys number: column_names and key_names' \
+          if @key_names.length != @column_names.length
+
+    @primary_key_in_event = @key_names[0]
+    ## Initialize Azure Search client Instance
+    @client=AzureSearch::Client::new( @endpoint, @api_key )
+
+    buffer_initialize(
+        :max_items => @flush_items,
+        :max_interval => @flush_interval_time,
+        :logger => @logger
+      )
+
+  end # def register
+
+  public
+  def receive(event)
+    # Simply save an event for later delivery
+    buffer_receive(event)
+  end # def event 
+
+  # called from Stud::Buffer#buffer_flush when there are events to flush
+  public
+  def flush (events, close=false)
+
+    documents = []  #this is the array of hashes to add Azure search
+    events.each do |event|
+      document = {}
+      event_hash = event.to_hash()
+
+      ## Check if event contains primary item that should be stored as 
+      ## primary key in Azure Search
+      if not event_hash.include?(@primary_key_in_event)
+        $logger.warn( "The event does not contain primary item!!: " + (event_hash.to_json).to_s)
+        next
+      end
+
+      @column_names.each_with_index do|k, i|
+        ikey = @key_names[i]
+        ival = event_hash.include?(ikey) ? event_hash[ikey] : ''
+        document[k] = ival
+      end
+      documents.push(document)
+    end
+
+    ## Skip in case there are no candidate documents to deliver
+    if documents.length < 1
+      return
+    end
+
+    begin
+      @client.add_documents(@search_index, documents)
+    rescue RestClient::ExceptionWithResponse => rcex
+      exdict = JSON.parse(rcex.response)
+      $logger.error("RestClient Error: '#{rcex.response}', data=>" + (documents.to_json).to_s)
+    rescue => ex
+      $logger.error( "Error: '#{ex}'" + ", data=>" + (documents.to_json).to_s)
+    end
+
+  end # def flush
+
+end # class LogStash::Outputs::AzureSearch

data/lib/logstash/outputs/azuresearch/client.rb

@@ -0,0 +1,37 @@
+require_relative 'constants'
+
+module AzureSearch
+  class Client
+    def initialize (api_url, api_key, api_version=AzureSearch::API_VERSION)
+      require 'rest-client'
+      require 'json'
+      @api_url = api_url
+      @api_version = api_version
+      @headers = {
+        'Content-Type' => "application/json; charset=UTF-8",
+        'Api-Key' => api_key,
+        'Accept' => "application/json",
+        'Accept-Charset' => "UTF-8"
+      }
+    end 
+
+    def add_documents(index_name, documents, merge=true)
+      raise ConfigError, 'no index_name' if index_name.empty?
+      raise ConfigError, 'no documents' if documents.empty?
+      action = merge ? 'mergeOrUpload' : 'upload' 
+      for document in documents
+        document['@search.action'] = action
+      end
+      req_body =  { :value => documents }.to_json
+      # p "REQ_BODY= #{req_body}"
+      # p "URI= #{@api_url}/indexes/#{index_name}/docs/index?api-version=#{@api_version}"
+      res = RestClient.post(
+            "#{@api_url}/indexes/#{index_name}/docs/index?api-version=#{@api_version}",
+            req_body,
+            @headers)
+      res
+    end
+
+  end
+end
+

data/lib/logstash/outputs/azuresearch/constants.rb

@@ -0,0 +1,5 @@
+module AzureSearch
+  API_VERSION = '2015-02-28'.freeze
+  MAX_DOCS_PER_INDEX_UPLOAD = 1000.freeze
+end
+

data/logstash-output-azuresearch.gemspec

@@ -0,0 +1,25 @@
+Gem::Specification.new do |s|
+  s.name = 'logstash-output-azuresearch'
+  s.version    =  File.read("VERSION").strip
+  s.authors = ["Yoichi Kawasaki"]
+  s.email = "yoichi.kawasaki@outlook.com"
+  s.summary = %q{logstash output plugin to store events into Azure Search}
+  s.description = s.summary
+  s.homepage = "http://github.com/yokawasa/logstash-output-azuresearch"
+  s.licenses = ["Apache License (2.0)"]
+  s.require_paths = ["lib"]
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT', 'VERSION']
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "output" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "rest-client"
+  s.add_runtime_dependency "logstash-core", ">= 2.0.0", "< 3.0.0"
+  s.add_runtime_dependency "logstash-codec-plain"
+  s.add_development_dependency "logstash-devutils"
+end

data/spec/outputs/azuresearch_spec.rb

@@ -0,0 +1,46 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/outputs/azuresearch"
+require "logstash/codecs/plain"
+require "logstash/event"
+
+describe LogStash::Outputs::Azuresearch do
+
+  let(:endpoint) { 'https://<YOUR ACCOUNT>.search.windows.net' }
+  let(:api_key) { '<AZURESEARCH API KEY>' }
+  let(:search_index) { '<SEARCH INDEX NAME>' }
+  let(:column_names) { ['id','user_name','message','created_at'] }
+  let(:key_names) { ['postid','user','content','posttime'] }
+
+  let(:azuresearch_config) {
+    { 
+      "endpoint" => endpoint, 
+      "api_key" => api_key,
+      "search_index" => search_index,
+      "column_names" => column_names,
+      "key_names" => key_names
+    }
+  }
+
+  let(:azuresearch_output) { LogStash::Outputs::Azuresearch.new(azuresearch_config) }
+
+  before do
+     azuresearch_output.register
+  end 
+
+  describe "#flush" do
+    it "Should successfully send the event to azuresearch" do
+      events = []
+      properties1 = { "postid" => "a0001", "user" => "foo", "content" => "msg0001", "posttime"=>"2016-12-27T00:01:00Z" }
+      properties2 = { "postid" => "a0002", "user" => "bar", "content" => "msg0002", "posttime"=>"2016-12-27T00:02:00Z" }
+      event1 =  LogStash::Event.new(properties1) 
+      event2 =  LogStash::Event.new(properties2) 
+      azuresearch_output.receive(event1)
+      azuresearch_output.receive(event2)
+      events.push(event1)
+      events.push(event2)
+      expect {azuresearch_output.flush(events)}.to_not raise_error
+    end
+  end
+
+end

metadata

@@ -0,0 +1,119 @@
+--- !ruby/object:Gem::Specification
+name: logstash-output-azuresearch
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Yoichi Kawasaki
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2016-12-29 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: rest-client
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  name: logstash-core
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-codec-plain
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: logstash output plugin to store events into Azure Search
+email: yoichi.kawasaki@outlook.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- Gemfile
+- LICENSE
+- README.md
+- VERSION
+- lib/logstash/outputs/azuresearch.rb
+- lib/logstash/outputs/azuresearch/client.rb
+- lib/logstash/outputs/azuresearch/constants.rb
+- logstash-output-azuresearch.gemspec
+- spec/outputs/azuresearch_spec.rb
+homepage: http://github.com/yokawasa/logstash-output-azuresearch
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: output
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.4.8
+signing_key:
+specification_version: 4
+summary: logstash output plugin to store events into Azure Search
+test_files:
+- spec/outputs/azuresearch_spec.rb