logstash-output-solr-lw 0.4.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/CHANGELOG.md +17 -0
- data/CONTRIBUTORS +11 -0
- data/DEVELOPER.md +2 -0
- data/Gemfile +2 -0
- data/LICENSE +13 -0
- data/README.md +187 -0
- data/lib/logstash/outputs/solr.rb +142 -0
- data/logstash-output-solr-lw.gemspec +33 -0
- data/spec/outputs/solr_spec.rb +240 -0
- metadata +209 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: d6da74296645506a69741e57a51604b8a89e6c945a563abb84a6b494e873004b
|
4
|
+
data.tar.gz: 06f349ca9b8013f4d6fb81bc9910ec4129465e8e3f4988315365e7b369fb2add
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 4a4530a90826245c3878b66472c2bd5ede427714b5a7231d4d155f0e45aa48757d0137d64936a317ab5a5f2f8f891ae696f874cffd0bf6badf0f12ac36b6c029
|
7
|
+
data.tar.gz: 86a9cb046c6df0d1a61f25c6d7d8d037a3056798c1d644a84e86590988c203ce014dc3edb1649844d75727a5f29779a34edf4f2e2297fc914d66556eccff500d
|
data/CHANGELOG.md
ADDED
@@ -0,0 +1,17 @@
|
|
1
|
+
## 0.4.0
|
2
|
+
- New feature to obtain Solr collection name from event's field value
|
3
|
+
- Boolean configuration property to control on whether commit every batch of events
|
4
|
+
- New commitWithin config property
|
5
|
+
- Remove feature that enforces Solr schema to avoid excessive Schema API requests
|
6
|
+
|
7
|
+
## 0.3.0
|
8
|
+
- Update rsolr-cloud.
|
9
|
+
|
10
|
+
## 0.2.0
|
11
|
+
- Bug fix.
|
12
|
+
|
13
|
+
## 0.2.0
|
14
|
+
- Supports SolrCloud.
|
15
|
+
|
16
|
+
## 0.1.0
|
17
|
+
- First release.
|
data/CONTRIBUTORS
ADDED
@@ -0,0 +1,11 @@
|
|
1
|
+
The following is a list of people who have contributed ideas, code, bug
|
2
|
+
reports, or in general have helped logstash along its way.
|
3
|
+
|
4
|
+
Contributors:
|
5
|
+
* Minoru Osuka (minoru.osuka@gmail.com)
|
6
|
+
* Alexey Serba (aserba@gmail.com)
|
7
|
+
|
8
|
+
Note: If you've sent us patches, bug reports, or otherwise contributed to
|
9
|
+
Logstash, and you aren't on the list above and want to be, please let us know
|
10
|
+
and we'll make sure you're here. Contributions from folks like you are what make
|
11
|
+
open source awesome.
|
data/DEVELOPER.md
ADDED
data/Gemfile
ADDED
data/LICENSE
ADDED
@@ -0,0 +1,13 @@
|
|
1
|
+
Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
|
2
|
+
|
3
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
4
|
+
you may not use this file except in compliance with the License.
|
5
|
+
You may obtain a copy of the License at
|
6
|
+
|
7
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
8
|
+
|
9
|
+
Unless required by applicable law or agreed to in writing, software
|
10
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
11
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
12
|
+
See the License for the specific language governing permissions and
|
13
|
+
limitations under the License.
|
data/README.md
ADDED
@@ -0,0 +1,187 @@
|
|
1
|
+
# Logstash Plugin
|
2
|
+
|
3
|
+
This is a plugin for [Logstash](https://github.com/elastic/logstash). It support [SolrCloud](https://lucene.apache.org/solr/guide/7_6/solrcloud.html), not only Standalone Solr.
|
4
|
+
|
5
|
+
Forked from [mosuka/logstash-output-solr](https://github.com/mosuka/logstash-output-solr).
|
6
|
+
|
7
|
+
It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
|
8
|
+
|
9
|
+
## Config parameters
|
10
|
+
|
11
|
+
### url
|
12
|
+
|
13
|
+
The server url for standalone Solr (for example http://localhost:8983/solr/collection1).
|
14
|
+
|
15
|
+
```
|
16
|
+
url http://localhost:8983/solr/collection1
|
17
|
+
```
|
18
|
+
|
19
|
+
### zk_host
|
20
|
+
|
21
|
+
The ZooKeeper connection string that SolrCloud refers to (for example localhost:2181/solr).
|
22
|
+
|
23
|
+
```
|
24
|
+
zk_host localhost:2181/solr
|
25
|
+
```
|
26
|
+
|
27
|
+
### collection
|
28
|
+
|
29
|
+
The SolrCloud collection name (default collection1).
|
30
|
+
|
31
|
+
```
|
32
|
+
collection collection1
|
33
|
+
```
|
34
|
+
|
35
|
+
### collection_field
|
36
|
+
|
37
|
+
A field name with the name of collection to send events to. If this property is nil (the default) then `collection` property is used. This feature allows to have a single Logstash pipeline that does Solr indexing into multiple Solr collections and is useful for [Pipeline-to-Pipeline Communication](https://www.elastic.co/guide/en/logstash/master/pipeline-to-pipeline.html)).
|
38
|
+
|
39
|
+
```
|
40
|
+
collection_field collection
|
41
|
+
```
|
42
|
+
|
43
|
+
### commit
|
44
|
+
|
45
|
+
Commit every batch of events? (default false)
|
46
|
+
|
47
|
+
```
|
48
|
+
commit true
|
49
|
+
```
|
50
|
+
|
51
|
+
### commitWithin
|
52
|
+
|
53
|
+
Defines how often (in ms) you want your events to be visible in Solr (default 10000, i.e. 10 seconds). For more information please see [commitWithin docs](https://lucene.apache.org/solr/guide/7_6/updatehandlers-in-solrconfig.html#UpdateHandlersinSolrConfig-Commits)
|
54
|
+
|
55
|
+
```
|
56
|
+
commitWithin 10000
|
57
|
+
```
|
58
|
+
|
59
|
+
### unique_key_field
|
60
|
+
|
61
|
+
A field name of unique key in the Solr schema.xml (default id).
|
62
|
+
|
63
|
+
```
|
64
|
+
unique_key_field id
|
65
|
+
```
|
66
|
+
|
67
|
+
### timestamp_field
|
68
|
+
|
69
|
+
A field name of event timestamp in the Solr schema.xml (default timestamp_tdt).
|
70
|
+
|
71
|
+
```
|
72
|
+
timestamp_field timestamp_tdt
|
73
|
+
```
|
74
|
+
|
75
|
+
### flush_size
|
76
|
+
|
77
|
+
A number of events to queue up before writing to Solr (default 100).
|
78
|
+
|
79
|
+
```
|
80
|
+
flush_size 100
|
81
|
+
```
|
82
|
+
|
83
|
+
## Plugin setup examples
|
84
|
+
|
85
|
+
### Send events to standalone Solr.
|
86
|
+
```
|
87
|
+
output {
|
88
|
+
solr {
|
89
|
+
url => "http://localhost:8983/solr/collection1"
|
90
|
+
}
|
91
|
+
}
|
92
|
+
|
93
|
+
```
|
94
|
+
|
95
|
+
### Send events to SolrCloud cluster.
|
96
|
+
```
|
97
|
+
output {
|
98
|
+
solr {
|
99
|
+
zk_host => "localhost:2181/solr"
|
100
|
+
collection => "collection1"
|
101
|
+
}
|
102
|
+
}
|
103
|
+
```
|
104
|
+
|
105
|
+
## Need Help?
|
106
|
+
|
107
|
+
Need help? Try #logstash on freenode IRC or the [https://discuss.elastic.co/c/logstash](https://discuss.elastic.co/c/logstash) discussion forum.
|
108
|
+
|
109
|
+
## Developing
|
110
|
+
|
111
|
+
### 1. Plugin Developement and Testing
|
112
|
+
|
113
|
+
#### Code
|
114
|
+
- To get started, you'll need JRuby with the Bundler gem installed.
|
115
|
+
|
116
|
+
- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
|
117
|
+
|
118
|
+
- Install dependencies
|
119
|
+
```sh
|
120
|
+
bundle install
|
121
|
+
```
|
122
|
+
|
123
|
+
#### Test
|
124
|
+
|
125
|
+
- Update your dependencies
|
126
|
+
|
127
|
+
```sh
|
128
|
+
bundle install
|
129
|
+
```
|
130
|
+
|
131
|
+
- Run tests
|
132
|
+
|
133
|
+
```sh
|
134
|
+
bundle exec rspec
|
135
|
+
```
|
136
|
+
|
137
|
+
### 2. Running your unpublished Plugin in Logstash
|
138
|
+
|
139
|
+
#### 2.1 Run in a local Logstash clone
|
140
|
+
|
141
|
+
- Edit Logstash `Gemfile` and add the local plugin path, for example:
|
142
|
+
|
143
|
+
```ruby
|
144
|
+
gem "logstash-output-solr-lw", :path => "/your/local/logstash-output-solr-lw"
|
145
|
+
```
|
146
|
+
|
147
|
+
- Install plugin
|
148
|
+
|
149
|
+
```sh
|
150
|
+
bin/plugin install --no-verify
|
151
|
+
```
|
152
|
+
|
153
|
+
- Run Logstash with your plugin
|
154
|
+
|
155
|
+
```sh
|
156
|
+
bin/logstash -e 'output {solr {}}'
|
157
|
+
```
|
158
|
+
|
159
|
+
At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
|
160
|
+
|
161
|
+
#### 2.2 Run in an installed Logstash
|
162
|
+
|
163
|
+
You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
|
164
|
+
|
165
|
+
- Build your plugin gem
|
166
|
+
|
167
|
+
```sh
|
168
|
+
gem build logstash-output-solr-lw.gemspec
|
169
|
+
```
|
170
|
+
|
171
|
+
- Install the plugin from the Logstash home
|
172
|
+
|
173
|
+
```sh
|
174
|
+
bin/plugin install /your/local/plugin/logstash-output-solr-lw.gem
|
175
|
+
```
|
176
|
+
|
177
|
+
- Start Logstash and proceed to test the plugin
|
178
|
+
|
179
|
+
## Contributing
|
180
|
+
|
181
|
+
All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
|
182
|
+
|
183
|
+
Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
|
184
|
+
|
185
|
+
It is more important to the community that you are able to contribute.
|
186
|
+
|
187
|
+
For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
|
@@ -0,0 +1,142 @@
|
|
1
|
+
# encoding: utf-8
|
2
|
+
require "logstash/outputs/base"
|
3
|
+
require "logstash/namespace"
|
4
|
+
|
5
|
+
require 'securerandom'
|
6
|
+
require "stud/buffer"
|
7
|
+
require 'rsolr'
|
8
|
+
require 'zk'
|
9
|
+
require 'rsolr/cloud'
|
10
|
+
|
11
|
+
# An Solr output that send data to Apache Solr.
|
12
|
+
class LogStash::Outputs::Solr < LogStash::Outputs::Base
|
13
|
+
config_name "solr"
|
14
|
+
|
15
|
+
include Stud::Buffer
|
16
|
+
|
17
|
+
# The Solr server url (for example http://localhost:8983/solr/collection1).
|
18
|
+
config :url, :validate => :string, :default => nil
|
19
|
+
|
20
|
+
# The ZooKeeper connection string that SolrCloud refers to (for example localhost:2181/solr).
|
21
|
+
config :zk_host, :validate => :string, :default => nil
|
22
|
+
# The SolrCloud collection name.
|
23
|
+
config :collection, :validate => :string, :default => 'collection1'
|
24
|
+
|
25
|
+
# A field name with the name of collection to send document to.
|
26
|
+
# If this property is nil then "collection" property is used.
|
27
|
+
# This allows Logstash multiple pipelines to send documents to a single
|
28
|
+
# pipeline that does Solr indexing into different Solr collections.
|
29
|
+
config :collection_field, :validate => :string, :default => nil
|
30
|
+
|
31
|
+
# Commit every batch?
|
32
|
+
config :commit, :validate => :boolean, :default => false
|
33
|
+
|
34
|
+
# Solr commitWithin parameter
|
35
|
+
config :commitWithin, :validate => :number, :default => 10000
|
36
|
+
|
37
|
+
# A field name of unique key in the Solr schema.xml (default id)
|
38
|
+
config :unique_key_field, :validate => :string, :default => 'id'
|
39
|
+
|
40
|
+
# A field name of event timestamp in the Solr schema.xml (default event_timestamp).
|
41
|
+
config :timestamp_field, :validate => :string, :default => 'timestamp_tdt'
|
42
|
+
|
43
|
+
# The batch size used in update.
|
44
|
+
config :flush_size, :validate => :number, :default => 100
|
45
|
+
|
46
|
+
# The batch size used in update.
|
47
|
+
config :idle_flush_time, :validate => :number, :default => 10
|
48
|
+
|
49
|
+
MODE_STANDALONE = 'Standalone'
|
50
|
+
MODE_SOLRCLOUD = 'SolrCloud'
|
51
|
+
|
52
|
+
public
|
53
|
+
def register
|
54
|
+
@mode = nil
|
55
|
+
if ! @url.nil? then
|
56
|
+
@mode = MODE_STANDALONE
|
57
|
+
elsif ! @zk_host.nil?
|
58
|
+
@mode = MODE_SOLRCLOUD
|
59
|
+
end
|
60
|
+
|
61
|
+
@solr_std = {}
|
62
|
+
@solr_cloud = nil
|
63
|
+
@zk = nil
|
64
|
+
|
65
|
+
if @mode == MODE_STANDALONE then
|
66
|
+
@solr_std[@collection] = RSolr.connect :url => @url
|
67
|
+
elsif @mode == MODE_SOLRCLOUD then
|
68
|
+
@zk = ZK.new(@zk_host)
|
69
|
+
cloud_connection = RSolr::Cloud::Connection.new(@zk)
|
70
|
+
@solr_cloud = RSolr::Client.new(cloud_connection, read_timeout: 60, open_timeout: 60)
|
71
|
+
end
|
72
|
+
|
73
|
+
buffer_initialize(
|
74
|
+
:max_items => @flush_size,
|
75
|
+
:max_interval => @idle_flush_time,
|
76
|
+
:logger => @logger
|
77
|
+
)
|
78
|
+
end # def register
|
79
|
+
|
80
|
+
public
|
81
|
+
def receive(event)
|
82
|
+
buffer_receive(event)
|
83
|
+
end # def event
|
84
|
+
|
85
|
+
public
|
86
|
+
def flush(events, close=false)
|
87
|
+
documents_per_col = {}
|
88
|
+
|
89
|
+
events.each do |event|
|
90
|
+
document = event.to_hash()
|
91
|
+
|
92
|
+
unless document.has_key?(@unique_key) then
|
93
|
+
document.merge!({@unique_key => SecureRandom.uuid})
|
94
|
+
end
|
95
|
+
|
96
|
+
unless document.has_key?(@timestamp_field) then
|
97
|
+
document.merge!({@timestamp_field => document['@timestamp']})
|
98
|
+
end
|
99
|
+
|
100
|
+
@logger.debug 'Record: %s' % document.inspect
|
101
|
+
|
102
|
+
collection = @collection
|
103
|
+
if @collection_field and document.has_key?(@collection_field) then
|
104
|
+
collection = document[@collection_field]
|
105
|
+
document.delete(@collection_field)
|
106
|
+
end
|
107
|
+
|
108
|
+
documents = documents_per_col.fetch(collection, [])
|
109
|
+
documents.push(document)
|
110
|
+
documents_per_col[collection] = documents
|
111
|
+
end
|
112
|
+
|
113
|
+
params = {}
|
114
|
+
if @commit
|
115
|
+
params[:commit] = true
|
116
|
+
end
|
117
|
+
params[:commitWithin] = @commitWithin
|
118
|
+
|
119
|
+
documents_per_col.each do |collection, documents|
|
120
|
+
if @mode == MODE_STANDALONE then
|
121
|
+
collection_url = @url.rpartition('/')[0] + '/' + collection
|
122
|
+
@solr_std[collection] ||= RSolr.connect :url => collection_url
|
123
|
+
@solr_std[collection].add documents, :params => params
|
124
|
+
@logger.info 'Added %d document(s) to Solr at "%s"' % [documents.count, collection_url]
|
125
|
+
elsif @mode == MODE_SOLRCLOUD then
|
126
|
+
@solr_cloud.add documents, collection: collection, :params => params
|
127
|
+
@logger.info 'Added %d document(s) to "%s" collection' % [documents.count, collection]
|
128
|
+
end
|
129
|
+
end
|
130
|
+
|
131
|
+
rescue Exception => e
|
132
|
+
@logger.warn("An error occurred while indexing", :exception => e.inspect)
|
133
|
+
end # def flush
|
134
|
+
|
135
|
+
public
|
136
|
+
def close
|
137
|
+
unless @zk.nil? then
|
138
|
+
@zk.close
|
139
|
+
end
|
140
|
+
end # def close
|
141
|
+
|
142
|
+
end # class LogStash::Outputs::Solr
|
@@ -0,0 +1,33 @@
|
|
1
|
+
Gem::Specification.new do |s|
|
2
|
+
s.name = 'logstash-output-solr-lw'
|
3
|
+
s.version = "0.4.0"
|
4
|
+
s.licenses = ["Apache License (2.0)"]
|
5
|
+
s.summary = "Logstash output plugin for sending data to Solr."
|
6
|
+
s.description = "Logstash output plugin for sending data to Solr. It supports SolrCloud, not only Standalone Solr."
|
7
|
+
s.authors = ["Minoru Osuka", "Alexey Serba"]
|
8
|
+
s.email = "aserba@gmail.com"
|
9
|
+
s.homepage = "https://github.com/lucidworks/logstash-output-solr"
|
10
|
+
s.require_paths = ["lib"]
|
11
|
+
|
12
|
+
# Files
|
13
|
+
s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE']
|
14
|
+
# Tests
|
15
|
+
s.test_files = s.files.grep(%r{^(test|spec|features)/})
|
16
|
+
|
17
|
+
# Special flag to let us know this is actually a logstash plugin
|
18
|
+
s.metadata = { "logstash_plugin" => "true", "logstash_group" => "output" }
|
19
|
+
|
20
|
+
# Gem dependencies
|
21
|
+
s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
|
22
|
+
s.add_runtime_dependency "logstash-codec-plain"
|
23
|
+
|
24
|
+
s.add_runtime_dependency 'rsolr', '~> 1.1.2'
|
25
|
+
s.add_runtime_dependency 'zk'
|
26
|
+
s.add_runtime_dependency 'rsolr-cloud-lw'
|
27
|
+
s.add_runtime_dependency 'stud'
|
28
|
+
|
29
|
+
s.add_development_dependency "logstash-core", ">= 1.60", "<= 2.99"
|
30
|
+
s.add_development_dependency "logstash-devutils"
|
31
|
+
s.add_development_dependency 'rake', '~> 10.5.0'
|
32
|
+
s.add_development_dependency 'zk-server', '~> 1.1.8'
|
33
|
+
end
|
@@ -0,0 +1,240 @@
|
|
1
|
+
# encoding: utf-8
|
2
|
+
require "logstash/devutils/rspec/spec_helper"
|
3
|
+
require "logstash/outputs/solr"
|
4
|
+
require "logstash/codecs/plain"
|
5
|
+
require "logstash/event"
|
6
|
+
|
7
|
+
require 'zk-server'
|
8
|
+
require 'zk'
|
9
|
+
|
10
|
+
describe LogStash::Outputs::Solr do
|
11
|
+
|
12
|
+
before do
|
13
|
+
@zk_server = nil
|
14
|
+
end
|
15
|
+
|
16
|
+
describe 'configuration' do
|
17
|
+
let(:config) {
|
18
|
+
{
|
19
|
+
'url' => 'http://localhost:8983/solr/collection1',
|
20
|
+
'zk_host' => 'localhost:2181/solr',
|
21
|
+
'collection' => 'collection1',
|
22
|
+
'flush_size' => 100,
|
23
|
+
'commit' => false,
|
24
|
+
'commitWithin' => 10000
|
25
|
+
}
|
26
|
+
}
|
27
|
+
|
28
|
+
it 'url' do
|
29
|
+
output = LogStash::Outputs::Solr.new(config)
|
30
|
+
expect(output.config['url']).to eq('http://localhost:8983/solr/collection1')
|
31
|
+
end
|
32
|
+
|
33
|
+
it 'zk_host' do
|
34
|
+
output = LogStash::Outputs::Solr.new(config)
|
35
|
+
expect(output.config['zk_host']).to eq('localhost:2181/solr')
|
36
|
+
end
|
37
|
+
|
38
|
+
it 'collection' do
|
39
|
+
output = LogStash::Outputs::Solr.new(config)
|
40
|
+
expect(output.config['collection']).to eq('collection1')
|
41
|
+
end
|
42
|
+
|
43
|
+
it 'flush_size' do
|
44
|
+
output = LogStash::Outputs::Solr.new(config)
|
45
|
+
expect(output.config['flush_size']).to eq(100)
|
46
|
+
end
|
47
|
+
|
48
|
+
it 'commit' do
|
49
|
+
output = LogStash::Outputs::Solr.new(config)
|
50
|
+
expect(output.config['commit']).to eq(false)
|
51
|
+
end
|
52
|
+
|
53
|
+
it 'commitWithin' do
|
54
|
+
output = LogStash::Outputs::Solr.new(config)
|
55
|
+
expect(output.config['commitWithin']).to eq(10000)
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
59
|
+
describe 'register_standalone' do
|
60
|
+
let(:config) {
|
61
|
+
{
|
62
|
+
'url' => 'http://localhost:8983/solr/collection1',
|
63
|
+
'flush_size' => 100
|
64
|
+
}
|
65
|
+
}
|
66
|
+
|
67
|
+
it 'mode' do
|
68
|
+
output = LogStash::Outputs::Solr.new(config)
|
69
|
+
output.register
|
70
|
+
|
71
|
+
mode = output.instance_variable_get('@mode')
|
72
|
+
expect(mode).to eq('Standalone')
|
73
|
+
end
|
74
|
+
end
|
75
|
+
|
76
|
+
describe 'register_solrcloud' do
|
77
|
+
before do
|
78
|
+
start_zookeeper
|
79
|
+
end
|
80
|
+
|
81
|
+
after do
|
82
|
+
stop_zookeeper
|
83
|
+
end
|
84
|
+
|
85
|
+
let(:config) {
|
86
|
+
{
|
87
|
+
'zk_host' => 'localhost:3292/solr',
|
88
|
+
'collection' => 'collection1',
|
89
|
+
'flush_size' => 100
|
90
|
+
}
|
91
|
+
}
|
92
|
+
|
93
|
+
it 'mode' do
|
94
|
+
output = LogStash::Outputs::Solr.new(config)
|
95
|
+
output.register
|
96
|
+
|
97
|
+
mode = output.instance_variable_get('@mode')
|
98
|
+
expect(mode).to eq('SolrCloud')
|
99
|
+
end
|
100
|
+
end
|
101
|
+
|
102
|
+
describe 'receive_standalone' do
|
103
|
+
let(:config) {
|
104
|
+
{
|
105
|
+
'url' => 'http://localhost:8983/solr/collection1',
|
106
|
+
'flush_size' => 100
|
107
|
+
}
|
108
|
+
}
|
109
|
+
|
110
|
+
let(:sample_record) {
|
111
|
+
{
|
112
|
+
'id' => 'change.me',
|
113
|
+
'title' => 'change.me'
|
114
|
+
}
|
115
|
+
}
|
116
|
+
|
117
|
+
it 'receive' do
|
118
|
+
output = LogStash::Outputs::Solr.new(config)
|
119
|
+
output.register
|
120
|
+
|
121
|
+
output.receive(sample_record)
|
122
|
+
end
|
123
|
+
end
|
124
|
+
|
125
|
+
describe 'receive_solrcloud' do
|
126
|
+
before do
|
127
|
+
start_zookeeper
|
128
|
+
end
|
129
|
+
|
130
|
+
after do
|
131
|
+
stop_zookeeper
|
132
|
+
end
|
133
|
+
|
134
|
+
let(:config) {
|
135
|
+
{
|
136
|
+
'zk_host' => 'localhost:3292/solr',
|
137
|
+
'collection' => 'collection1',
|
138
|
+
'flush_size' => 100
|
139
|
+
}
|
140
|
+
}
|
141
|
+
|
142
|
+
let(:sample_record) {
|
143
|
+
{
|
144
|
+
'id' => 'change.me',
|
145
|
+
'title' => 'change.me'
|
146
|
+
}
|
147
|
+
}
|
148
|
+
|
149
|
+
it 'receive' do
|
150
|
+
output = LogStash::Outputs::Solr.new(config)
|
151
|
+
output.register
|
152
|
+
|
153
|
+
output.receive(sample_record)
|
154
|
+
end
|
155
|
+
end
|
156
|
+
|
157
|
+
describe 'multiple_collections' do
|
158
|
+
before do
|
159
|
+
start_zookeeper
|
160
|
+
end
|
161
|
+
|
162
|
+
after do
|
163
|
+
stop_zookeeper
|
164
|
+
end
|
165
|
+
|
166
|
+
let(:config) {
|
167
|
+
{
|
168
|
+
'zk_host' => 'localhost:3292/solr',
|
169
|
+
'collection_field' => 'collection',
|
170
|
+
'flush_size' => 100
|
171
|
+
}
|
172
|
+
}
|
173
|
+
|
174
|
+
let(:sample_record1) {
|
175
|
+
{
|
176
|
+
'id' => 'test1',
|
177
|
+
'collection' => 'col1'
|
178
|
+
}
|
179
|
+
}
|
180
|
+
let(:sample_record2) {
|
181
|
+
{
|
182
|
+
'id' => 'test2',
|
183
|
+
'collection' => 'col2'
|
184
|
+
}
|
185
|
+
}
|
186
|
+
let(:sample_record3) {
|
187
|
+
{
|
188
|
+
'id' => 'test3'
|
189
|
+
}
|
190
|
+
}
|
191
|
+
|
192
|
+
it 'receive' do
|
193
|
+
output = LogStash::Outputs::Solr.new(config)
|
194
|
+
output.register
|
195
|
+
|
196
|
+
output.receive(sample_record1)
|
197
|
+
output.receive(sample_record2)
|
198
|
+
output.receive(sample_record3)
|
199
|
+
end
|
200
|
+
end
|
201
|
+
|
202
|
+
def start_zookeeper
|
203
|
+
@zk_server = ZK::Server.new do |config|
|
204
|
+
config.client_port = 3292
|
205
|
+
config.enable_jmx = true
|
206
|
+
config.force_sync = false
|
207
|
+
end
|
208
|
+
|
209
|
+
@zk_server.run
|
210
|
+
|
211
|
+
zk = ZK.new('localhost:3292')
|
212
|
+
delete_nodes(zk, '/solr')
|
213
|
+
create_nodes(zk, '/solr/live_nodes')
|
214
|
+
['localhost:8983_solr'].each do |node|
|
215
|
+
zk.create("/solr/live_nodes/#{node}", '', mode: :ephemeral)
|
216
|
+
end
|
217
|
+
end
|
218
|
+
|
219
|
+
def stop_zookeeper
|
220
|
+
@zk_server.shutdown
|
221
|
+
end
|
222
|
+
|
223
|
+
def delete_nodes(zk, path)
|
224
|
+
zk.children(path).each do |node|
|
225
|
+
delete_nodes(zk, File.join(path, node))
|
226
|
+
end
|
227
|
+
zk.delete(path)
|
228
|
+
rescue ZK::Exceptions::NoNode
|
229
|
+
end
|
230
|
+
|
231
|
+
def create_nodes(zk, path)
|
232
|
+
parent_path = File.dirname(path)
|
233
|
+
unless zk.exists?(parent_path, :watch => true) then
|
234
|
+
create_nodes(zk, parent_path)
|
235
|
+
end
|
236
|
+
zk.create(path)
|
237
|
+
end
|
238
|
+
end
|
239
|
+
|
240
|
+
|
metadata
ADDED
@@ -0,0 +1,209 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: logstash-output-solr-lw
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.4.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Minoru Osuka
|
8
|
+
- Alexey Serba
|
9
|
+
autorequire:
|
10
|
+
bindir: bin
|
11
|
+
cert_chain: []
|
12
|
+
date: 2019-01-24 00:00:00.000000000 Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - ">="
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '1.60'
|
20
|
+
- - "<="
|
21
|
+
- !ruby/object:Gem::Version
|
22
|
+
version: '2.99'
|
23
|
+
name: logstash-core-plugin-api
|
24
|
+
prerelease: false
|
25
|
+
type: :runtime
|
26
|
+
version_requirements: !ruby/object:Gem::Requirement
|
27
|
+
requirements:
|
28
|
+
- - ">="
|
29
|
+
- !ruby/object:Gem::Version
|
30
|
+
version: '1.60'
|
31
|
+
- - "<="
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '2.99'
|
34
|
+
- !ruby/object:Gem::Dependency
|
35
|
+
requirement: !ruby/object:Gem::Requirement
|
36
|
+
requirements:
|
37
|
+
- - ">="
|
38
|
+
- !ruby/object:Gem::Version
|
39
|
+
version: '0'
|
40
|
+
name: logstash-codec-plain
|
41
|
+
prerelease: false
|
42
|
+
type: :runtime
|
43
|
+
version_requirements: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - ">="
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '0'
|
48
|
+
- !ruby/object:Gem::Dependency
|
49
|
+
requirement: !ruby/object:Gem::Requirement
|
50
|
+
requirements:
|
51
|
+
- - "~>"
|
52
|
+
- !ruby/object:Gem::Version
|
53
|
+
version: 1.1.2
|
54
|
+
name: rsolr
|
55
|
+
prerelease: false
|
56
|
+
type: :runtime
|
57
|
+
version_requirements: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - "~>"
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: 1.1.2
|
62
|
+
- !ruby/object:Gem::Dependency
|
63
|
+
requirement: !ruby/object:Gem::Requirement
|
64
|
+
requirements:
|
65
|
+
- - ">="
|
66
|
+
- !ruby/object:Gem::Version
|
67
|
+
version: '0'
|
68
|
+
name: zk
|
69
|
+
prerelease: false
|
70
|
+
type: :runtime
|
71
|
+
version_requirements: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - ">="
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '0'
|
76
|
+
- !ruby/object:Gem::Dependency
|
77
|
+
requirement: !ruby/object:Gem::Requirement
|
78
|
+
requirements:
|
79
|
+
- - ">="
|
80
|
+
- !ruby/object:Gem::Version
|
81
|
+
version: '0'
|
82
|
+
name: rsolr-cloud-lw
|
83
|
+
prerelease: false
|
84
|
+
type: :runtime
|
85
|
+
version_requirements: !ruby/object:Gem::Requirement
|
86
|
+
requirements:
|
87
|
+
- - ">="
|
88
|
+
- !ruby/object:Gem::Version
|
89
|
+
version: '0'
|
90
|
+
- !ruby/object:Gem::Dependency
|
91
|
+
requirement: !ruby/object:Gem::Requirement
|
92
|
+
requirements:
|
93
|
+
- - ">="
|
94
|
+
- !ruby/object:Gem::Version
|
95
|
+
version: '0'
|
96
|
+
name: stud
|
97
|
+
prerelease: false
|
98
|
+
type: :runtime
|
99
|
+
version_requirements: !ruby/object:Gem::Requirement
|
100
|
+
requirements:
|
101
|
+
- - ">="
|
102
|
+
- !ruby/object:Gem::Version
|
103
|
+
version: '0'
|
104
|
+
- !ruby/object:Gem::Dependency
|
105
|
+
requirement: !ruby/object:Gem::Requirement
|
106
|
+
requirements:
|
107
|
+
- - ">="
|
108
|
+
- !ruby/object:Gem::Version
|
109
|
+
version: '1.60'
|
110
|
+
- - "<="
|
111
|
+
- !ruby/object:Gem::Version
|
112
|
+
version: '2.99'
|
113
|
+
name: logstash-core
|
114
|
+
prerelease: false
|
115
|
+
type: :development
|
116
|
+
version_requirements: !ruby/object:Gem::Requirement
|
117
|
+
requirements:
|
118
|
+
- - ">="
|
119
|
+
- !ruby/object:Gem::Version
|
120
|
+
version: '1.60'
|
121
|
+
- - "<="
|
122
|
+
- !ruby/object:Gem::Version
|
123
|
+
version: '2.99'
|
124
|
+
- !ruby/object:Gem::Dependency
|
125
|
+
requirement: !ruby/object:Gem::Requirement
|
126
|
+
requirements:
|
127
|
+
- - ">="
|
128
|
+
- !ruby/object:Gem::Version
|
129
|
+
version: '0'
|
130
|
+
name: logstash-devutils
|
131
|
+
prerelease: false
|
132
|
+
type: :development
|
133
|
+
version_requirements: !ruby/object:Gem::Requirement
|
134
|
+
requirements:
|
135
|
+
- - ">="
|
136
|
+
- !ruby/object:Gem::Version
|
137
|
+
version: '0'
|
138
|
+
- !ruby/object:Gem::Dependency
|
139
|
+
requirement: !ruby/object:Gem::Requirement
|
140
|
+
requirements:
|
141
|
+
- - "~>"
|
142
|
+
- !ruby/object:Gem::Version
|
143
|
+
version: 10.5.0
|
144
|
+
name: rake
|
145
|
+
prerelease: false
|
146
|
+
type: :development
|
147
|
+
version_requirements: !ruby/object:Gem::Requirement
|
148
|
+
requirements:
|
149
|
+
- - "~>"
|
150
|
+
- !ruby/object:Gem::Version
|
151
|
+
version: 10.5.0
|
152
|
+
- !ruby/object:Gem::Dependency
|
153
|
+
requirement: !ruby/object:Gem::Requirement
|
154
|
+
requirements:
|
155
|
+
- - "~>"
|
156
|
+
- !ruby/object:Gem::Version
|
157
|
+
version: 1.1.8
|
158
|
+
name: zk-server
|
159
|
+
prerelease: false
|
160
|
+
type: :development
|
161
|
+
version_requirements: !ruby/object:Gem::Requirement
|
162
|
+
requirements:
|
163
|
+
- - "~>"
|
164
|
+
- !ruby/object:Gem::Version
|
165
|
+
version: 1.1.8
|
166
|
+
description: Logstash output plugin for sending data to Solr. It supports SolrCloud,
|
167
|
+
not only Standalone Solr.
|
168
|
+
email: aserba@gmail.com
|
169
|
+
executables: []
|
170
|
+
extensions: []
|
171
|
+
extra_rdoc_files: []
|
172
|
+
files:
|
173
|
+
- CHANGELOG.md
|
174
|
+
- CONTRIBUTORS
|
175
|
+
- DEVELOPER.md
|
176
|
+
- Gemfile
|
177
|
+
- LICENSE
|
178
|
+
- README.md
|
179
|
+
- lib/logstash/outputs/solr.rb
|
180
|
+
- logstash-output-solr-lw.gemspec
|
181
|
+
- spec/outputs/solr_spec.rb
|
182
|
+
homepage: https://github.com/lucidworks/logstash-output-solr
|
183
|
+
licenses:
|
184
|
+
- Apache License (2.0)
|
185
|
+
metadata:
|
186
|
+
logstash_plugin: 'true'
|
187
|
+
logstash_group: output
|
188
|
+
post_install_message:
|
189
|
+
rdoc_options: []
|
190
|
+
require_paths:
|
191
|
+
- lib
|
192
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
193
|
+
requirements:
|
194
|
+
- - ">="
|
195
|
+
- !ruby/object:Gem::Version
|
196
|
+
version: '0'
|
197
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
198
|
+
requirements:
|
199
|
+
- - ">="
|
200
|
+
- !ruby/object:Gem::Version
|
201
|
+
version: '0'
|
202
|
+
requirements: []
|
203
|
+
rubyforge_project:
|
204
|
+
rubygems_version: 2.6.13
|
205
|
+
signing_key:
|
206
|
+
specification_version: 4
|
207
|
+
summary: Logstash output plugin for sending data to Solr.
|
208
|
+
test_files:
|
209
|
+
- spec/outputs/solr_spec.rb
|