logstash-output-azuresearch 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: f31f4188a841a0cfd75d86b5f4cd893b59f26abe
4
+ data.tar.gz: d78ab9f954ac2cb9fa1f2ea640631b81b7ba0ee1
5
+ SHA512:
6
+ metadata.gz: cad4a079a66b3f01514c0d4825349415de6341e857c1e27315b8cc38333887d629c02e732a645118921ba22043e49ea25c34213d13745b52dff6a5dc5fbf0468
7
+ data.tar.gz: 418f7b3bad65f7b8fe9e0b1b81475a06746142044e0d7bfaa5c7a7974dad7e14c43822d7d2262dd86138aba99b8bbed5b254243228eb08522ef9d2a8b72041a9
data/CHANGELOG.md ADDED
@@ -0,0 +1,3 @@
1
+ ## 0.1.0
2
+
3
+ * Inital Release
data/CONTRIBUTORS ADDED
@@ -0,0 +1,10 @@
1
+ The following is a list of people who have contributed ideas, code, bug
2
+ reports, or in general have helped logstash along its way.
3
+
4
+ Contributors:
5
+ * Yoichi Kawasaki (yokawasa)
6
+
7
+ Note: If you've sent us patches, bug reports, or otherwise contributed to
8
+ Logstash, and you aren't on the list above and want to be, please let us know
9
+ and we'll make sure you're here. Contributions from folks like you are what make
10
+ open source awesome.
data/Gemfile ADDED
@@ -0,0 +1,2 @@
1
+ source 'https://rubygems.org'
2
+ gemspec
data/LICENSE ADDED
@@ -0,0 +1,13 @@
1
+ Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
2
+
3
+ Licensed under the Apache License, Version 2.0 (the "License");
4
+ you may not use this file except in compliance with the License.
5
+ You may obtain a copy of the License at
6
+
7
+ http://www.apache.org/licenses/LICENSE-2.0
8
+
9
+ Unless required by applicable law or agreed to in writing, software
10
+ distributed under the License is distributed on an "AS IS" BASIS,
11
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ See the License for the specific language governing permissions and
13
+ limitations under the License.
data/README.md ADDED
@@ -0,0 +1,94 @@
1
+ # Azure Search output plugin for Logstash
2
+
3
+ logstash-output-azuresearch is a logstash plugin to output to Azure Search. [Logstash](https://www.elastic.co/products/logstash) is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite [destinations](https://www.elastic.co/products/logstash). [Azure Search](https://docs.microsoft.com/en-us/azure/search/search-what-is-azure-search) is a managed cloud search service provided in Microsoft Azure.
4
+
5
+ ## Installation
6
+
7
+ You can install this plugin using the Logstash "plugin" or "logstash-plugin" (for newer versions of Logstash) command:
8
+ ```
9
+ bin/plugin install logstash-output-azuresearch
10
+ # or
11
+ bin/logstash-plugin install logstash-output-azuresearch (Newer versions of Logstash)
12
+ ```
13
+ Please see [Logstash reference](https://www.elastic.co/guide/en/logstash/current/offline-plugins.html) for more information.
14
+
15
+ ## Configuration
16
+
17
+ ```
18
+ output {
19
+ azuresearch {
20
+ endpoint => "https://<YOUR ACCOUNT>.search.windows.net"
21
+ api_key => "<AZURESEARCH API KEY>"
22
+ search_index => "<SEARCH INDEX NAME>"
23
+ column_names => ['col1','col2','col3'..] ## ## list of column names (array)
24
+ key_names => ['key1','key2','key3'..] ## list of Key names (array)
25
+ flush_items => <FLUSH_ITEMS_NUM>
26
+ flush_interval_time => <FLUSH INTERVAL TIME(sec)>
27
+ }
28
+ }
29
+ ```
30
+
31
+ * **endpoint (required)** - Azure Search service endpoint URI
32
+ * **api\_key (required)** - Azure Search API key
33
+ * **search\_index (required)** - Azure Search Index name to insert records
34
+ * **column\_names (required)** - List of column names (array) in a target Azure search index. 1st item in column_names should be primary key
35
+ * **key\_names (optional)** - Default:[] (empty array). List of key names (array) in in-coming record to insert. The number of keys in key_names must be equal to the one of columns in column_names. Also the order or each item in key_names must match the one of each items in column_names.
36
+ * **flush_items (optional)** - Default 50. Max number of items to buffer before flushing (1 - 1000).
37
+ * **flush_interval_time (optional)** - Default 5. Max number of seconds to wait between flushes.
38
+
39
+ ## Tests
40
+
41
+ Here is an example configuration where Logstash's event source and destination are configured as standard input and Azure Search respectively.
42
+
43
+ ### Example Configuration
44
+ ```
45
+ input {
46
+ stdin {
47
+ codec => json_lines
48
+ }
49
+ }
50
+
51
+ output {
52
+ azuresearch {
53
+ endpoint => "https://<YOUR ACCOUNT>.search.windows.net"
54
+ api_key => "<AZURESEARCH API KEY>"
55
+ search_index => "<SEARCH INDEX NAME>"
56
+ column_names => ['id','user_name','message','created_at']
57
+ key_names => ['postid','user','content','posttime']
58
+ flush_items => 100
59
+ flush_interval_time => 5
60
+ }
61
+ }
62
+ ```
63
+ You can find example configuration files in logstash-output-azuresearch/examples.
64
+
65
+ ### Run the plugin with the example configuration
66
+
67
+ Now you run logstash with the the example configuration like this:
68
+ ```
69
+ # Test your logstash configuration before actually running the logstash
70
+ bin/logstash -f logstash-stdin-json-to-azuresearch.conf --configtest
71
+ # run
72
+ bin/logstash -f logstash-stdin-json-to-azuresearch.conf
73
+ ```
74
+
75
+ Here is an expected output for sample input (JSON Lines):
76
+
77
+ <u>JSON Lines</u>
78
+ ```
79
+ { "id": "a001", "user_name": "user001", "message":"msg001", "created_at":"2016-12-28T00:01:00Z" },
80
+ { "id": "a002", "user_name": "user002", "message":"msg002", "created_at":"2016-12-28T00:02:00Z" },
81
+ { "id": "a003", "user_name": "user003", "message":"msg003", "created_at":"2016-12-28T00:03:00Z" },
82
+ ```
83
+ <u>Output (Azure Search POST message)</u>
84
+ ```
85
+ "value": [
86
+ { "@search.score": 1, "id": "a001", "user_name": "user001", "message": "msg001", "created_at": "2016-12-28T00:01:00Z" },
87
+ { "@search.score": 1, "id": "a002", "user_name": "user002", "message": "msg002", "created_at": "2016-12-28T00:02:00Z" },
88
+ { "@search.score": 1, "id": "a003", "user_name": "user003", "message": "msg003", "created_at": "2016-12-28T00:03:00Z" }
89
+ ]
90
+ ```
91
+
92
+ ## Contributing
93
+
94
+ Bug reports and pull requests are welcome on GitHub at https://github.com/yokawasa/logstash-output-azuresearch.
data/VERSION ADDED
@@ -0,0 +1 @@
1
+ 0.1.0
@@ -0,0 +1,105 @@
1
+ # encoding: utf-8
2
+ require "logstash/outputs/base"
3
+ require "logstash/namespace"
4
+ require "stud/buffer"
5
+
6
+ class LogStash::Outputs::Azuresearch < LogStash::Outputs::Base
7
+ include Stud::Buffer
8
+
9
+ config_name "azuresearch"
10
+
11
+ # Azure Search Endpoint URL
12
+ config :endpoint, :validate => :string, :required => true
13
+
14
+ # zure Search API key
15
+ config :api_key, :validate => :string, :required => true
16
+
17
+ # Azure Search Index name to insert records
18
+ config :search_index, :validate => :string, :required => true
19
+
20
+ # Column names in a target Azure search index.
21
+ # 1st item in column_names should be primary key
22
+ config :column_names, :validate => :array, :required => true
23
+
24
+ # Key names in incomming record to insert.
25
+ # The number of keys in key_names must be equal to the one of columns in column_names.
26
+ # Also the order or each item in key_names must match the one of each items in column_names.
27
+ config :key_names, :validate => :array, :default => []
28
+
29
+ # Max number of items to buffer before flushing (1 - 1000). Default 50.
30
+ config :flush_items, :validate => :number, :default => 50
31
+
32
+ # Max number of seconds to wait between flushes. Default 5
33
+ config :flush_interval_time, :validate => :number, :default => 5
34
+
35
+
36
+ public
37
+ def register
38
+ require_relative 'azuresearch/client'
39
+
40
+ ## Configure
41
+ if @key_names.length < 1
42
+ @key_names = @column_names
43
+ end
44
+ raise ArgumentError, 'NOT match keys number: column_names and key_names' \
45
+ if @key_names.length != @column_names.length
46
+
47
+ @primary_key_in_event = @key_names[0]
48
+ ## Initialize Azure Search client Instance
49
+ @client=AzureSearch::Client::new( @endpoint, @api_key )
50
+
51
+ buffer_initialize(
52
+ :max_items => @flush_items,
53
+ :max_interval => @flush_interval_time,
54
+ :logger => @logger
55
+ )
56
+
57
+ end # def register
58
+
59
+ public
60
+ def receive(event)
61
+ # Simply save an event for later delivery
62
+ buffer_receive(event)
63
+ end # def event
64
+
65
+ # called from Stud::Buffer#buffer_flush when there are events to flush
66
+ public
67
+ def flush (events, close=false)
68
+
69
+ documents = [] #this is the array of hashes to add Azure search
70
+ events.each do |event|
71
+ document = {}
72
+ event_hash = event.to_hash()
73
+
74
+ ## Check if event contains primary item that should be stored as
75
+ ## primary key in Azure Search
76
+ if not event_hash.include?(@primary_key_in_event)
77
+ $logger.warn( "The event does not contain primary item!!: " + (event_hash.to_json).to_s)
78
+ next
79
+ end
80
+
81
+ @column_names.each_with_index do|k, i|
82
+ ikey = @key_names[i]
83
+ ival = event_hash.include?(ikey) ? event_hash[ikey] : ''
84
+ document[k] = ival
85
+ end
86
+ documents.push(document)
87
+ end
88
+
89
+ ## Skip in case there are no candidate documents to deliver
90
+ if documents.length < 1
91
+ return
92
+ end
93
+
94
+ begin
95
+ @client.add_documents(@search_index, documents)
96
+ rescue RestClient::ExceptionWithResponse => rcex
97
+ exdict = JSON.parse(rcex.response)
98
+ $logger.error("RestClient Error: '#{rcex.response}', data=>" + (documents.to_json).to_s)
99
+ rescue => ex
100
+ $logger.error( "Error: '#{ex}'" + ", data=>" + (documents.to_json).to_s)
101
+ end
102
+
103
+ end # def flush
104
+
105
+ end # class LogStash::Outputs::AzureSearch
@@ -0,0 +1,37 @@
1
+ require_relative 'constants'
2
+
3
+ module AzureSearch
4
+ class Client
5
+ def initialize (api_url, api_key, api_version=AzureSearch::API_VERSION)
6
+ require 'rest-client'
7
+ require 'json'
8
+ @api_url = api_url
9
+ @api_version = api_version
10
+ @headers = {
11
+ 'Content-Type' => "application/json; charset=UTF-8",
12
+ 'Api-Key' => api_key,
13
+ 'Accept' => "application/json",
14
+ 'Accept-Charset' => "UTF-8"
15
+ }
16
+ end
17
+
18
+ def add_documents(index_name, documents, merge=true)
19
+ raise ConfigError, 'no index_name' if index_name.empty?
20
+ raise ConfigError, 'no documents' if documents.empty?
21
+ action = merge ? 'mergeOrUpload' : 'upload'
22
+ for document in documents
23
+ document['@search.action'] = action
24
+ end
25
+ req_body = { :value => documents }.to_json
26
+ # p "REQ_BODY= #{req_body}"
27
+ # p "URI= #{@api_url}/indexes/#{index_name}/docs/index?api-version=#{@api_version}"
28
+ res = RestClient.post(
29
+ "#{@api_url}/indexes/#{index_name}/docs/index?api-version=#{@api_version}",
30
+ req_body,
31
+ @headers)
32
+ res
33
+ end
34
+
35
+ end
36
+ end
37
+
@@ -0,0 +1,5 @@
1
+ module AzureSearch
2
+ API_VERSION = '2015-02-28'.freeze
3
+ MAX_DOCS_PER_INDEX_UPLOAD = 1000.freeze
4
+ end
5
+
@@ -0,0 +1,25 @@
1
+ Gem::Specification.new do |s|
2
+ s.name = 'logstash-output-azuresearch'
3
+ s.version = File.read("VERSION").strip
4
+ s.authors = ["Yoichi Kawasaki"]
5
+ s.email = "yoichi.kawasaki@outlook.com"
6
+ s.summary = %q{logstash output plugin to store events into Azure Search}
7
+ s.description = s.summary
8
+ s.homepage = "http://github.com/yokawasa/logstash-output-azuresearch"
9
+ s.licenses = ["Apache License (2.0)"]
10
+ s.require_paths = ["lib"]
11
+
12
+ # Files
13
+ s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT', 'VERSION']
14
+ # Tests
15
+ s.test_files = s.files.grep(%r{^(test|spec|features)/})
16
+
17
+ # Special flag to let us know this is actually a logstash plugin
18
+ s.metadata = { "logstash_plugin" => "true", "logstash_group" => "output" }
19
+
20
+ # Gem dependencies
21
+ s.add_runtime_dependency "rest-client"
22
+ s.add_runtime_dependency "logstash-core", ">= 2.0.0", "< 3.0.0"
23
+ s.add_runtime_dependency "logstash-codec-plain"
24
+ s.add_development_dependency "logstash-devutils"
25
+ end
@@ -0,0 +1,46 @@
1
+ # encoding: utf-8
2
+ require "logstash/devutils/rspec/spec_helper"
3
+ require "logstash/outputs/azuresearch"
4
+ require "logstash/codecs/plain"
5
+ require "logstash/event"
6
+
7
+ describe LogStash::Outputs::Azuresearch do
8
+
9
+ let(:endpoint) { 'https://<YOUR ACCOUNT>.search.windows.net' }
10
+ let(:api_key) { '<AZURESEARCH API KEY>' }
11
+ let(:search_index) { '<SEARCH INDEX NAME>' }
12
+ let(:column_names) { ['id','user_name','message','created_at'] }
13
+ let(:key_names) { ['postid','user','content','posttime'] }
14
+
15
+ let(:azuresearch_config) {
16
+ {
17
+ "endpoint" => endpoint,
18
+ "api_key" => api_key,
19
+ "search_index" => search_index,
20
+ "column_names" => column_names,
21
+ "key_names" => key_names
22
+ }
23
+ }
24
+
25
+ let(:azuresearch_output) { LogStash::Outputs::Azuresearch.new(azuresearch_config) }
26
+
27
+ before do
28
+ azuresearch_output.register
29
+ end
30
+
31
+ describe "#flush" do
32
+ it "Should successfully send the event to azuresearch" do
33
+ events = []
34
+ properties1 = { "postid" => "a0001", "user" => "foo", "content" => "msg0001", "posttime"=>"2016-12-27T00:01:00Z" }
35
+ properties2 = { "postid" => "a0002", "user" => "bar", "content" => "msg0002", "posttime"=>"2016-12-27T00:02:00Z" }
36
+ event1 = LogStash::Event.new(properties1)
37
+ event2 = LogStash::Event.new(properties2)
38
+ azuresearch_output.receive(event1)
39
+ azuresearch_output.receive(event2)
40
+ events.push(event1)
41
+ events.push(event2)
42
+ expect {azuresearch_output.flush(events)}.to_not raise_error
43
+ end
44
+ end
45
+
46
+ end
metadata ADDED
@@ -0,0 +1,119 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: logstash-output-azuresearch
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - Yoichi Kawasaki
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2016-12-29 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ requirement: !ruby/object:Gem::Requirement
15
+ requirements:
16
+ - - ">="
17
+ - !ruby/object:Gem::Version
18
+ version: '0'
19
+ name: rest-client
20
+ prerelease: false
21
+ type: :runtime
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - ">="
25
+ - !ruby/object:Gem::Version
26
+ version: '0'
27
+ - !ruby/object:Gem::Dependency
28
+ requirement: !ruby/object:Gem::Requirement
29
+ requirements:
30
+ - - ">="
31
+ - !ruby/object:Gem::Version
32
+ version: 2.0.0
33
+ - - "<"
34
+ - !ruby/object:Gem::Version
35
+ version: 3.0.0
36
+ name: logstash-core
37
+ prerelease: false
38
+ type: :runtime
39
+ version_requirements: !ruby/object:Gem::Requirement
40
+ requirements:
41
+ - - ">="
42
+ - !ruby/object:Gem::Version
43
+ version: 2.0.0
44
+ - - "<"
45
+ - !ruby/object:Gem::Version
46
+ version: 3.0.0
47
+ - !ruby/object:Gem::Dependency
48
+ requirement: !ruby/object:Gem::Requirement
49
+ requirements:
50
+ - - ">="
51
+ - !ruby/object:Gem::Version
52
+ version: '0'
53
+ name: logstash-codec-plain
54
+ prerelease: false
55
+ type: :runtime
56
+ version_requirements: !ruby/object:Gem::Requirement
57
+ requirements:
58
+ - - ">="
59
+ - !ruby/object:Gem::Version
60
+ version: '0'
61
+ - !ruby/object:Gem::Dependency
62
+ requirement: !ruby/object:Gem::Requirement
63
+ requirements:
64
+ - - ">="
65
+ - !ruby/object:Gem::Version
66
+ version: '0'
67
+ name: logstash-devutils
68
+ prerelease: false
69
+ type: :development
70
+ version_requirements: !ruby/object:Gem::Requirement
71
+ requirements:
72
+ - - ">="
73
+ - !ruby/object:Gem::Version
74
+ version: '0'
75
+ description: logstash output plugin to store events into Azure Search
76
+ email: yoichi.kawasaki@outlook.com
77
+ executables: []
78
+ extensions: []
79
+ extra_rdoc_files: []
80
+ files:
81
+ - CHANGELOG.md
82
+ - CONTRIBUTORS
83
+ - Gemfile
84
+ - LICENSE
85
+ - README.md
86
+ - VERSION
87
+ - lib/logstash/outputs/azuresearch.rb
88
+ - lib/logstash/outputs/azuresearch/client.rb
89
+ - lib/logstash/outputs/azuresearch/constants.rb
90
+ - logstash-output-azuresearch.gemspec
91
+ - spec/outputs/azuresearch_spec.rb
92
+ homepage: http://github.com/yokawasa/logstash-output-azuresearch
93
+ licenses:
94
+ - Apache License (2.0)
95
+ metadata:
96
+ logstash_plugin: 'true'
97
+ logstash_group: output
98
+ post_install_message:
99
+ rdoc_options: []
100
+ require_paths:
101
+ - lib
102
+ required_ruby_version: !ruby/object:Gem::Requirement
103
+ requirements:
104
+ - - ">="
105
+ - !ruby/object:Gem::Version
106
+ version: '0'
107
+ required_rubygems_version: !ruby/object:Gem::Requirement
108
+ requirements:
109
+ - - ">="
110
+ - !ruby/object:Gem::Version
111
+ version: '0'
112
+ requirements: []
113
+ rubyforge_project:
114
+ rubygems_version: 2.4.8
115
+ signing_key:
116
+ specification_version: 4
117
+ summary: logstash output plugin to store events into Azure Search
118
+ test_files:
119
+ - spec/outputs/azuresearch_spec.rb