logstash-output-dis 1.1.2 → 1.1.3

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: '019f0e86a61a2053ec8b9dbd53dddda42d57ed37'
4
- data.tar.gz: e62e3e0a3158a2fed3856badf0f83c1d465051bb
3
+ metadata.gz: 4ee98db25b7754948eff32f950777dd33ddd8347
4
+ data.tar.gz: 6f30bd7159cdf8cba37f5813748a79a579709c7b
5
5
  SHA512:
6
- metadata.gz: 0d836921059524dd1a991618aa56e9b676fa15a69d4e98a1175fda13e577e612b2b5e81d32f591cc2708aaaeb3f4897f7af69bbe6caa41b57f9c6423fd4af8ef
7
- data.tar.gz: cf32b4f30e9b47eed4c594ee9ffa23fc04ca0b57c9a49a54ba108a799c47990717d49a2bedec392fb58f8fa6c2d38a83e17f08c62c1dd3b02dcebcc4e3640af2
6
+ metadata.gz: 33d6c9ce17d5f5db825336742c25fda208d47d937bb265034f2d1131ac2632da5afec2e0b7c4a09f9648445c587b265f5c715aefdd0a7a949c88f959b11af0b8
7
+ data.tar.gz: e713b8279b696e49d70ea60b9ec4a8eaa8f92d89d2acc0c7a0e362f20e424272a0e62b18813ef889b1ed2bcab5f92e65ea874b1b21e602e871df21933b461890
data/Gemfile CHANGED
@@ -1,11 +1,11 @@
1
- source 'http://gems.ruby-china.com'
2
-
3
- gemspec
4
-
5
- logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
6
- use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
7
-
8
- if Dir.exist?(logstash_path) && use_logstash_source
9
- gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
10
- gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
11
- end
1
+ source 'http://gems.ruby-china.com'
2
+
3
+ gemspec
4
+
5
+ logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
6
+ use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
7
+
8
+ if Dir.exist?(logstash_path) && use_logstash_source
9
+ gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
10
+ gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
11
+ end
data/LICENSE CHANGED
@@ -1,13 +1,13 @@
1
- Copyright (c) 2012-2018 Elasticsearch <http://www.elastic.co>
2
-
3
- Licensed under the Apache License, Version 2.0 (the "License");
4
- you may not use this file except in compliance with the License.
5
- You may obtain a copy of the License at
6
-
7
- http://www.apache.org/licenses/LICENSE-2.0
8
-
9
- Unless required by applicable law or agreed to in writing, software
10
- distributed under the License is distributed on an "AS IS" BASIS,
11
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
- See the License for the specific language governing permissions and
13
- limitations under the License.
1
+ Copyright (c) 2012-2018 Elasticsearch <http://www.elastic.co>
2
+
3
+ Licensed under the Apache License, Version 2.0 (the "License");
4
+ you may not use this file except in compliance with the License.
5
+ You may obtain a copy of the License at
6
+
7
+ http://www.apache.org/licenses/LICENSE-2.0
8
+
9
+ Unless required by applicable law or agreed to in writing, software
10
+ distributed under the License is distributed on an "AS IS" BASIS,
11
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ See the License for the specific language governing permissions and
13
+ limitations under the License.
data/NOTICE.TXT CHANGED
@@ -1,5 +1,5 @@
1
- Elasticsearch
2
- Copyright 2012-2015 Elasticsearch
3
-
4
- This product includes software developed by The Apache Software
1
+ Elasticsearch
2
+ Copyright 2012-2015 Elasticsearch
3
+
4
+ This product includes software developed by The Apache Software
5
5
  Foundation (http://www.apache.org/).
data/README.md CHANGED
@@ -1,70 +1,66 @@
1
- # Logstash Output DIS
2
-
3
- This is a plugin for [Logstash](https://github.com/elastic/logstash). It will send log records to a DIS stream, using the DIS-Kafka-Adapter.
4
-
5
- ## Requirements
6
-
7
- To get started using this plugin, you will need three things:
8
-
9
- 1. JDK 1.8 +
10
- 2. JRuby with the Bundler gem installed, 9.0.0.0 +
11
- 3. Maven
12
- 4. Logstash, 6.0.0 to 6.1.0
13
-
14
- ## Installation
15
- 当前插件未发布到`RubyGems.org`,无法直接从`RubyGems.org`安装插件,只能从本地安装。
16
- ### 0. 修改 RubyGems 镜像地址
17
- gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/
18
-
19
- ### 0. 安装 dis-kafka-adapter
20
-
21
-
22
- ### 1. 安装 JRuby
23
- ### 2. 安装 Bundler gem
24
- gem install bundler
25
-
26
- ### 3. 安装依赖
27
- bundle install
28
- rake install_jars
29
- gem build logstash-output-dis.gemspec
30
-
31
- ### 4. 编辑 Logstash 的`Gemfile`,并添加本地插件路径
32
- gem "logstash-output-dis", :path => "/your/local/logstash-output-dis"
33
-
34
- ### 5. 安装插件到 Logstash
35
- bin/logstash-plugin install --no-verify
36
-
37
- ## Usage
38
-
39
- ```properties
40
- output
41
- {
42
- dis {
43
- stream => ["YOU_DIS_STREAM_NAME"]
44
- endpoint => "https://dis.cn-north-1.myhuaweicloud.com"
45
- ak => "YOU_ACCESS_KEY_ID"
46
- sk => "YOU_SECRET_KEY_ID"
47
- region => "cn-north-1"
48
- project_id => "YOU_PROJECT_ID"
49
- group_id => "YOU_GROUP_ID"
50
- decorate_events => true
51
- auto_offset_reset => "earliest"
52
- }
53
- }
54
- ```
55
-
56
- ## Configuration
57
-
58
- ### Parameters
59
-
60
- | Name | Description | Default |
61
- | :----------------------- | :--------------------------------------- | :--------------------------------------- |
62
- | stream | 指定在DIS服务上创建的通道名称。 | - |
63
- | ak | 用户的Access Key,可从华为云控制台“我的凭证”页获取。 | - |
64
- | sk | 用户的Secret Key,可从华为云控制台“我的凭证”页获取。 | - |
65
- | region | 将数据上传到指定Region的DIS服务。 | cn-north-1 |
66
- | project_id | 用户所属区域的项目ID,可从华为云控制台“我的凭证”页获取。 | - |
67
- | endpoint | DIS对应Region的数据接口地址。 | https://dis.cn-north-1.myhuaweicloud.com |
68
-
69
- ## License
1
+ # Logstash Output DIS
2
+
3
+ This is a plugin for [Logstash](https://github.com/elastic/logstash). It will send log records to a DIS stream, using the DIS-Kafka-Adapter.
4
+
5
+ ## Requirements
6
+
7
+ To get started using this plugin, you will need three things:
8
+
9
+ 1. JDK 1.8 +
10
+ 2. JRuby with the Bundler gem installed, 9.0.0.0 +
11
+ 3. Maven
12
+ 4. Logstash
13
+
14
+ ## Installation
15
+ ### 0. 修改 RubyGems 镜像地址
16
+ gem sources --add https://gems.ruby-china.com/ --remove https://rubygems.org/
17
+
18
+ ### 1. 安装 JRuby
19
+ ### 2. 安装 Bundler gem
20
+ gem install bundler
21
+
22
+ ### 3. 安装依赖
23
+ bundle install
24
+ rake install_jars
25
+ gem build logstash-output-dis.gemspec
26
+
27
+ ### 4. 编辑 Logstash 的`Gemfile`,并添加本地插件路径
28
+ gem "logstash-output-dis", :path => "/your/local/logstash-output-dis"
29
+
30
+ ### 5. 安装插件到 Logstash
31
+ bin/logstash-plugin install --no-verify
32
+
33
+ ## Usage
34
+
35
+ ```properties
36
+ output
37
+ {
38
+ dis {
39
+ stream => ["YOU_DIS_STREAM_NAME"]
40
+ endpoint => "https://dis.cn-north-1.myhuaweicloud.com"
41
+ ak => "YOU_ACCESS_KEY_ID"
42
+ sk => "YOU_SECRET_KEY_ID"
43
+ region => "cn-north-1"
44
+ project_id => "YOU_PROJECT_ID"
45
+ group_id => "YOU_GROUP_ID"
46
+ decorate_events => true
47
+ auto_offset_reset => "earliest"
48
+ }
49
+ }
50
+ ```
51
+
52
+ ## Configuration
53
+
54
+ ### Parameters
55
+
56
+ | Name | Description | Default |
57
+ | :----------------------- | :--------------------------------------- | :--------------------------------------- |
58
+ | stream | 指定在DIS服务上创建的通道名称。 | - |
59
+ | ak | 用户的Access Key,可从华为云控制台“我的凭证”页获取。 | - |
60
+ | sk | 用户的Secret Key,可从华为云控制台“我的凭证”页获取。 | - |
61
+ | region | 将数据上传到指定Region的DIS服务。 | cn-north-1 |
62
+ | project_id | 用户所属区域的项目ID,可从华为云控制台“我的凭证”页获取。 | - |
63
+ | endpoint | DIS对应Region的数据接口地址。 | https://dis.cn-north-1.myhuaweicloud.com |
64
+
65
+ ## License
70
66
  [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0.html)
@@ -1,318 +1,303 @@
1
- require 'logstash/namespace'
2
- require 'logstash/outputs/base'
3
- require 'java'
4
- require 'logstash-output-dis_jars.rb'
5
-
6
- java_import com.huaweicloud.dis.adapter.kafka.clients.producer.ProducerRecord
7
-
8
- # Write events to a DIS stream, using DIS Kafka Adapter.
9
- class LogStash::Outputs::Dis < LogStash::Outputs::Base
10
- declare_threadsafe!
11
-
12
- config_name 'dis'
13
-
14
- default :codec, 'plain'
15
-
16
- config :default_trusted_jks_enabled, :validate => :boolean, :default => false
17
- config :security_token, :validate => :string
18
- config :exception_retries, :validate => :number, :default => 8
19
- config :records_retries, :validate => :number, :default => 20
20
- config :proxy_host, :validate => :string
21
- config :proxy_port, :validate => :number, :default => 80
22
- config :proxy_protocol, :validate => ["http", "https"], :default => "http"
23
- config :proxy_username, :validate => :string
24
- config :proxy_password, :validate => :string
25
- config :proxy_workstation, :validate => :string
26
- config :proxy_domain, :validate => :string
27
- config :proxy_non_proxy_hosts, :validate => :string
28
-
29
- # The producer will attempt to batch records together into fewer requests whenever multiple
30
- # records are being sent to the same partition. This helps performance on both the client
31
- # and the server. This configuration controls the default batch size in bytes.
32
- config :batch_size, :validate => :number, :default => 16384
33
- config :batch_count, :validate => :number, :default => 5000
34
-
35
- # The total bytes of memory the producer can use to buffer records waiting to be sent to the server.
36
- config :buffer_memory, :validate => :number, :default => 33554432
37
- config :buffer_count, :validate => :number, :default => 5000
38
- # The producer groups together any records that arrive in between request
39
- # transmissions into a single batched request. Normally this occurs only under
40
- # load when records arrive faster than they can be sent out. However in some circumstances
41
- # the client may want to reduce the number of requests even under moderate load.
42
- # This setting accomplishes this by adding a small amount of artificial delay—that is,
43
- # rather than immediately sending out a record the producer will wait for up to the given delay
44
- # to allow other records to be sent so that the sends can be batched together.
45
- config :linger_ms, :validate => :number, :default => 50
46
- config :block_on_buffer_full, :validate => :boolean, :default => false
47
- # block time when buffer is full
48
- config :max_block_ms, :validate => :number, :default => 60000
49
- # max wait time in single backoff
50
- config :backoff_max_interval_ms, :validate => :number, :default => 30000
51
- config :max_in_flight_requests_per_connection, :validate => :number, :default => 50
52
- config :records_retriable_error_code, :validate => :string, :default => "DIS.4303,DIS.5"
53
- config :order_by_partition, :validate => :boolean, :default => false
54
- config :metadata_timeout_ms, :validate => :number, :default => 600000
55
- # The key for the message
56
- config :message_key, :validate => :string
57
- config :partition_id, :validate => :string
58
- # the timeout setting for initial metadata request to fetch topic metadata.
59
- config :metadata_fetch_timeout_ms, :validate => :number, :default => 60000
60
- # the max time in milliseconds before a metadata refresh is forced.
61
- config :metadata_max_age_ms, :validate => :number, :default => 300000
62
- # The size of the TCP receive buffer to use when reading data
63
- config :receive_buffer_bytes, :validate => :number, :default => 32768
64
- # The configuration controls the maximum amount of time the client will wait
65
- # for the response of a request. If the response is not received before the timeout
66
- # elapses the client will resend the request if necessary or fail the request if
67
- # retries are exhausted.
68
- config :request_timeout_ms, :validate => :string
69
- # The default retry behavior is to retry until successful. To prevent data loss,
70
- # the use of this setting is discouraged.
71
- #
72
- # If you choose to set `retries`, a value greater than zero will cause the
73
- # client to only retry a fixed number of times. This will result in data loss
74
- # if a transient error outlasts your retry count.
75
- #
76
- # A value less than zero is a configuration error.
77
- config :retries, :validate => :number
78
- # The amount of time to wait before attempting to retry a failed produce request to a given topic partition.
79
- config :retry_backoff_ms, :validate => :number, :default => 100
80
-
81
-
82
- # The DIS stream to produce messages to
83
- config :stream, :validate => :string, :required => true
84
- # DIS Gateway endpoint
85
- config :endpoint, :validate => :string, :default => "https://dis.cn-north-1.myhuaweicloud.com"
86
- # The ProjectId of the specified region, it can be obtained from My Credential Page
87
- config :project_id, :validate => :string
88
- # Specifies use which region of DIS, now DIS only support cn-north-1
89
- config :region, :validate => :string, :default => "cn-north-1"
90
- # The Access Key ID for hwclouds, it can be obtained from My Credential Page
91
- config :ak, :validate => :string, :required => true
92
- # The Secret key ID is encrypted or not
93
- config :is_sk_encrypted, :default => false
94
- # The encrypt key used to encypt the Secret Key Id
95
- config :encrypt_key, :validate => :string
96
- # The Secret Key ID for hwclouds, it can be obtained from My Credential Page
97
- config :sk, :validate => :string, :required => true
98
- # Serializer class for the key of the message
99
- config :key_serializer, :validate => :string, :default => 'com.huaweicloud.dis.adapter.kafka.common.serialization.StringSerializer'
100
- # Serializer class for the value of the message
101
- config :value_serializer, :validate => :string, :default => 'com.huaweicloud.dis.adapter.kafka.common.serialization.StringSerializer'
102
-
103
- public
104
- def register
105
- @thread_batch_map = Concurrent::Hash.new
106
-
107
- if !@retries.nil?
108
- if @retries < 0
109
- raise ConfigurationError, "A negative retry count (#{@retries}) is not valid. Must be a value >= 0"
110
- end
111
-
112
- @logger.warn("Kafka output is configured with finite retry. This instructs Logstash to LOSE DATA after a set number of send attempts fails. If you do not want to lose data if Kafka is down, then you must remove the retry setting.", :retries => @retries)
113
- end
114
-
115
-
116
- @producer = create_producer
117
- if value_serializer == 'com.huaweicloud.dis.adapter.kafka.common.serialization.StringSerializer'
118
- @codec.on_event do |event, data|
119
- write_to_dis(event, data)
120
- end
121
- elsif value_serializer == 'com.huaweicloud.dis.adapter.kafka.common.serialization.ByteArraySerializer'
122
- @codec.on_event do |event, data|
123
- write_to_dis(event, data.to_java_bytes)
124
- end
125
- else
126
- raise ConfigurationError, "'value_serializer' only supports com.huaweicloud.dis.adapter.kafka.common.serialization.ByteArraySerializer and com.huaweicloud.dis.adapter.kafka.common.serialization.StringSerializer"
127
- end
128
- end
129
-
130
- # def register
131
-
132
- def prepare(record)
133
- # This output is threadsafe, so we need to keep a batch per thread.
134
- @thread_batch_map[Thread.current].add(record)
135
- end
136
-
137
- def multi_receive(events)
138
- t = Thread.current
139
- if !@thread_batch_map.include?(t)
140
- @thread_batch_map[t] = java.util.ArrayList.new(events.size)
141
- end
142
-
143
- events.each do |event|
144
- break if event == LogStash::SHUTDOWN
145
- @codec.encode(event)
146
- end
147
-
148
- batch = @thread_batch_map[t]
149
- if batch.any?
150
- retrying_send(batch)
151
- batch.clear
152
- end
153
- end
154
-
155
- def retrying_send(batch)
156
- remaining = @retries;
157
-
158
- while batch.any?
159
- if !remaining.nil?
160
- if remaining < 0
161
- # TODO(sissel): Offer to DLQ? Then again, if it's a transient fault,
162
- # DLQing would make things worse (you dlq data that would be successful
163
- # after the fault is repaired)
164
- logger.info("Exhausted user-configured retry count when sending to Kafka. Dropping these events.",
165
- :max_retries => @retries, :drop_count => batch.count)
166
- break
167
- end
168
-
169
- remaining -= 1
170
- end
171
-
172
- failures = []
173
-
174
- futures = batch.collect do |record|
175
- begin
176
- # send() can throw an exception even before the future is created.
177
- @producer.send(record)
178
- rescue org.apache.kafka.common.errors.TimeoutException => e
179
- failures << record
180
- nil
181
- rescue org.apache.kafka.common.errors.InterruptException => e
182
- failures << record
183
- nil
184
- rescue com.huaweicloud.dis.adapter.kafka.common.errors.SerializationException => e
185
- # TODO(sissel): Retrying will fail because the data itself has a problem serializing.
186
- # TODO(sissel): Let's add DLQ here.
187
- failures << record
188
- nil
189
- end
190
- end.compact
191
-
192
- futures.each_with_index do |future, i|
193
- begin
194
- result = future.get()
195
- rescue => e
196
- # TODO(sissel): Add metric to count failures, possibly by exception type.
197
- logger.warn("KafkaProducer.send() failed: #{e}", :exception => e)
198
- failures << batch[i]
199
- end
200
- end
201
-
202
- # No failures? Cool. Let's move on.
203
- break if failures.empty?
204
-
205
- # Otherwise, retry with any failed transmissions
206
- batch = failures
207
- delay = @retry_backoff_ms / 1000.0
208
- logger.info("Sending batch to DIS failed. Will retry after a delay.", :batch_size => batch.size,
209
- :failures => failures.size, :sleep => delay);
210
- sleep(delay)
211
- end
212
-
213
- end
214
-
215
- def close
216
- @producer.close
217
- end
218
-
219
- private
220
-
221
- def write_to_dis(event, serialized_data)
222
- stream = event.get("stream");
223
- if stream.nil?
224
- stream = @stream;
225
- end
226
-
227
- message_key = event.get("partition_key");
228
- if message_key.nil?
229
- message_key = @message_key;
230
- end
231
-
232
- partition_id = event.get("partition_id");
233
-
234
- if message_key.nil? && partition_id.nil?
235
- # record = ProducerRecord.new(event.sprintf(@stream), serialized_data)
236
- record = ProducerRecord.new(stream, serialized_data)
237
- elsif partition_id.nil?
238
- # record = ProducerRecord.new(event.sprintf(@stream), event.sprintf(@message_key), serialized_data)
239
- # record = ProducerRecord.new(stream, event.sprintf(@message_key), serialized_data)
240
- record = ProducerRecord.new(stream, message_key, serialized_data)
241
- else
242
- record = ProducerRecord.new(stream, partition_id.to_i, message_key, serialized_data)
243
- end
244
- prepare(record)
245
- rescue LogStash::ShutdownSignal
246
- @logger.debug('DIS Kafka producer got shutdown signal')
247
- rescue => e
248
- @logger.warn('DIS kafka producer threw exception, restarting',
249
- :exception => e)
250
- end
251
-
252
- def create_producer
253
- begin
254
- props = java.util.Properties.new
255
- kafka = com.huaweicloud.dis.adapter.kafka.clients.producer.ProducerConfig
256
-
257
- props.put("IS_DEFAULT_TRUSTED_JKS_ENABLED", default_trusted_jks_enabled.to_s)
258
- props.put("security.token", security_token) unless security_token.nil?
259
- props.put("exception.retries", exception_retries.to_s)
260
- props.put("records.retries", records_retries.to_s)
261
- props.put("PROXY_HOST", proxy_host) unless proxy_host.nil?
262
- props.put("PROXY_PORT", proxy_port.to_s)
263
- props.put("PROXY_PROTOCOL", proxy_protocol)
264
- props.put("PROXY_USERNAME", proxy_username) unless proxy_username.nil?
265
- props.put("PROXY_PASSWORD", proxy_password) unless proxy_password.nil?
266
- props.put("PROXY_WORKSTATION", proxy_workstation) unless proxy_workstation.nil?
267
- props.put("PROXY_DOMAIN", proxy_domain) unless proxy_domain.nil?
268
- props.put("NON_PROXY_HOSTS", proxy_non_proxy_hosts) unless proxy_non_proxy_hosts.nil?
269
-
270
- props.put("batch.size", batch_size.to_s)
271
- props.put("batch.count", batch_count.to_s)
272
- props.put("buffer.memory", buffer_memory.to_s)
273
- props.put("buffer.count", buffer_count.to_s)
274
- props.put("linger.ms", linger_ms.to_s)
275
- props.put("block.on.buffer.full", block_on_buffer_full.to_s)
276
- props.put("max.block.ms", max_block_ms.to_s)
277
- props.put("backoff.max.interval.ms", backoff_max_interval_ms.to_s)
278
- props.put("max.in.flight.requests.per.connection", max_in_flight_requests_per_connection.to_s)
279
- props.put("records.retriable.error.code", records_retriable_error_code) unless records_retriable_error_code.nil?
280
- props.put("order.by.partition", order_by_partition.to_s)
281
- props.put("metadata.timeout.ms", metadata_timeout_ms.to_s)
282
- # props.put(kafka::RETRIES_CONFIG, retries.to_s) unless retries.nil?
283
- # props.put(kafka::RETRY_BACKOFF_MS_CONFIG, retry_backoff_ms.to_s)
284
- props.put("key.deserializer", "com.huaweicloud.dis.adapter.kafka.common.serialization.StringDeserializer")
285
- props.put("value.deserializer", "com.huaweicloud.dis.adapter.kafka.common.serialization.StringDeserializer")
286
-
287
- # endpoint, project_id, region, ak, sk
288
- props.put("endpoint", endpoint)
289
- props.put("projectId", project_id)
290
- props.put("region", region)
291
- props.put("ak", ak)
292
- if is_sk_encrypted
293
- decrypted_sk = decrypt(@sk)
294
- props.put("sk", decrypted_sk)
295
- else
296
- props.put("sk", sk)
297
- end
298
-
299
-
300
- com.huaweicloud.dis.adapter.kafka.clients.producer.DISKafkaProducer.new(props)
301
- rescue => e
302
- logger.error("Unable to create DIS Kafka producer from given configuration",
303
- :kafka_error_message => e,
304
- :cause => e.respond_to?(:getCause) ? e.getCause() : nil)
305
- raise e
306
- end
307
- end
308
-
309
- private
310
- def decrypt(encrypted_sk)
311
- com.huaweicloud.dis.util.encrypt.EncryptUtils.dec([@encrypt_key].to_java(java.lang.String), encrypted_sk)
312
- rescue => e
313
- logger.error("Unable to decrypt sk from given configuration",
314
- :decrypt_error_message => e,
315
- :cause => e.respond_to?(:getCause) ? e.getCause() : nil)
316
- end
317
-
318
- end #class LogStash::Outputs::Dis
1
+ require 'logstash/namespace'
2
+ require 'logstash/outputs/base'
3
+ require 'java'
4
+ require 'logstash-output-dis_jars.rb'
5
+
6
+ java_import com.huaweicloud.dis.adapter.kafka.clients.producer.ProducerRecord
7
+
8
+ # Write events to a DIS stream, using DIS Kafka Adapter.
9
+ class LogStash::Outputs::Dis < LogStash::Outputs::Base
10
+ declare_threadsafe!
11
+
12
+ config_name 'dis'
13
+
14
+ default :codec, 'plain'
15
+
16
+ config :default_trusted_jks_enabled, :validate => :boolean, :default => false
17
+ config :security_token, :validate => :string
18
+ config :exception_retries, :validate => :number, :default => 8
19
+ config :records_retries, :validate => :number, :default => 20
20
+ config :proxy_host, :validate => :string
21
+ config :proxy_port, :validate => :number, :default => 80
22
+ config :proxy_protocol, :validate => ["http", "https"], :default => "http"
23
+ config :proxy_username, :validate => :string
24
+ config :proxy_password, :validate => :string
25
+ config :proxy_workstation, :validate => :string
26
+ config :proxy_domain, :validate => :string
27
+ config :proxy_non_proxy_hosts, :validate => :string
28
+
29
+ # The producer will attempt to batch records together into fewer requests whenever multiple
30
+ # records are being sent to the same partition. This helps performance on both the client
31
+ # and the server. This configuration controls the default batch size in bytes.
32
+ config :batch_size, :validate => :number, :default => 1048576
33
+ config :batch_count, :validate => :number, :default => 5000
34
+
35
+ # The total bytes of memory the producer can use to buffer records waiting to be sent to the server.
36
+ config :buffer_memory, :validate => :number, :default => 33554432
37
+ config :buffer_count, :validate => :number, :default => 5000
38
+ # The producer groups together any records that arrive in between request
39
+ # transmissions into a single batched request. Normally this occurs only under
40
+ # load when records arrive faster than they can be sent out. However in some circumstances
41
+ # the client may want to reduce the number of requests even under moderate load.
42
+ # This setting accomplishes this by adding a small amount of artificial delay—that is,
43
+ # rather than immediately sending out a record the producer will wait for up to the given delay
44
+ # to allow other records to be sent so that the sends can be batched together.
45
+ config :linger_ms, :validate => :number, :default => 50
46
+ config :block_on_buffer_full, :validate => :boolean, :default => false
47
+ # block time when buffer is full
48
+ config :max_block_ms, :validate => :number, :default => 60000
49
+ # max wait time in single backoff
50
+ config :backoff_max_interval_ms, :validate => :number, :default => 30000
51
+ config :max_in_flight_requests_per_connection, :validate => :number, :default => 50
52
+ config :records_retriable_error_code, :validate => :string, :default => "DIS.4303,DIS.5"
53
+ config :order_by_partition, :validate => :boolean, :default => false
54
+ config :body_serialize_type, :validate => :string, :default => "protobuf"
55
+ config :metadata_timeout_ms, :validate => :number, :default => 600000
56
+ # The key for the message
57
+ config :message_key, :validate => :string
58
+ config :partition_id, :validate => :string
59
+ # the timeout setting for initial metadata request to fetch topic metadata.
60
+ config :metadata_fetch_timeout_ms, :validate => :number, :default => 60000
61
+ # the max time in milliseconds before a metadata refresh is forced.
62
+ config :metadata_max_age_ms, :validate => :number, :default => 300000
63
+ # The size of the TCP receive buffer to use when reading data
64
+ config :receive_buffer_bytes, :validate => :number, :default => 32768
65
+ # The configuration controls the maximum amount of time the client will wait
66
+ # for the response of a request. If the response is not received before the timeout
67
+ # elapses the client will resend the request if necessary or fail the request if
68
+ # retries are exhausted.
69
+ config :request_timeout_ms, :validate => :string
70
+ # The default retry behavior is to retry until successful. To prevent data loss,
71
+ # the use of this setting is discouraged.
72
+ #
73
+ # If you choose to set `retries`, a value greater than zero will cause the
74
+ # client to only retry a fixed number of times. This will result in data loss
75
+ # if a transient error outlasts your retry count.
76
+ #
77
+ # A value less than zero is a configuration error.
78
+ config :retries, :validate => :number
79
+ # The amount of time to wait before attempting to retry a failed produce request to a given topic partition.
80
+ config :retry_backoff_ms, :validate => :number, :default => 100
81
+
82
+
83
+ # The DIS stream to produce messages to
84
+ config :stream, :validate => :string, :required => true
85
+ # DIS Gateway endpoint
86
+ config :endpoint, :validate => :string, :default => "https://dis.cn-north-1.myhuaweicloud.com"
87
+ # The ProjectId of the specified region, it can be obtained from My Credential Page
88
+ config :project_id, :validate => :string
89
+ # Specifies use which region of DIS, now DIS only support cn-north-1
90
+ config :region, :validate => :string, :default => "cn-north-1"
91
+ # The Access Key ID for hwclouds, it can be obtained from My Credential Page
92
+ config :ak, :validate => :string, :required => true
93
+ # The Secret key ID is encrypted or not
94
+ config :is_sk_encrypted, :default => false
95
+ # The encrypt key used to encypt the Secret Key Id
96
+ config :encrypt_key, :validate => :string
97
+ # The Secret Key ID for hwclouds, it can be obtained from My Credential Page
98
+ config :sk, :validate => :string, :required => true
99
+ # Serializer class for the key of the message
100
+ config :key_serializer, :validate => :string, :default => 'com.huaweicloud.dis.adapter.kafka.common.serialization.StringSerializer'
101
+ # Serializer class for the value of the message
102
+ config :value_serializer, :validate => :string, :default => 'com.huaweicloud.dis.adapter.kafka.common.serialization.StringSerializer'
103
+
104
+ public
105
+ def register
106
+ @thread_batch_map = Concurrent::Hash.new
107
+
108
+ if !@retries.nil?
109
+ if @retries < 0
110
+ raise ConfigurationError, "A negative retry count (#{@retries}) is not valid. Must be a value >= 0"
111
+ end
112
+
113
+ @logger.warn("Kafka output is configured with finite retry. This instructs Logstash to LOSE DATA after a set number of send attempts fails. If you do not want to lose data if Kafka is down, then you must remove the retry setting.", :retries => @retries)
114
+ end
115
+
116
+
117
+ @producer = create_producer
118
+ if value_serializer == 'com.huaweicloud.dis.adapter.kafka.common.serialization.StringSerializer'
119
+ @codec.on_event do |event, data|
120
+ write_to_dis(event, data)
121
+ end
122
+ elsif value_serializer == 'com.huaweicloud.dis.adapter.kafka.common.serialization.ByteArraySerializer'
123
+ @codec.on_event do |event, data|
124
+ write_to_dis(event, data.to_java_bytes)
125
+ end
126
+ else
127
+ raise ConfigurationError, "'value_serializer' only supports com.huaweicloud.dis.adapter.kafka.common.serialization.ByteArraySerializer and com.huaweicloud.dis.adapter.kafka.common.serialization.StringSerializer"
128
+ end
129
+ end
130
+
131
+ # def register
132
+
133
+ def prepare(record)
134
+ # This output is threadsafe, so we need to keep a batch per thread.
135
+ @thread_batch_map[Thread.current].add(record)
136
+ end
137
+
138
+ def multi_receive(events)
139
+ t = Thread.current
140
+ if !@thread_batch_map.include?(t)
141
+ @thread_batch_map[t] = java.util.ArrayList.new(events.size)
142
+ end
143
+
144
+ events.each do |event|
145
+ break if event == LogStash::SHUTDOWN
146
+ @codec.encode(event)
147
+ end
148
+
149
+ batch = @thread_batch_map[t]
150
+ if batch.any?
151
+ retrying_send(batch)
152
+ batch.clear
153
+ end
154
+ end
155
+
156
+ def retrying_send(batch)
157
+ remaining = @retries;
158
+
159
+ while batch.any?
160
+ if !remaining.nil?
161
+ if remaining < 0
162
+ # TODO(sissel): Offer to DLQ? Then again, if it's a transient fault,
163
+ # DLQing would make things worse (you dlq data that would be successful
164
+ # after the fault is repaired)
165
+ logger.info("Exhausted user-configured retry count when sending to Kafka. Dropping these events.",
166
+ :max_retries => @retries, :drop_count => batch.count)
167
+ break
168
+ end
169
+
170
+ remaining -= 1
171
+ end
172
+
173
+ failures = []
174
+
175
+ futures = batch.collect do |record|
176
+ begin
177
+ # send() can throw an exception even before the future is created.
178
+ @producer.send(record)
179
+ rescue org.apache.kafka.common.errors.TimeoutException => e
180
+ failures << record
181
+ nil
182
+ rescue org.apache.kafka.common.errors.InterruptException => e
183
+ failures << record
184
+ nil
185
+ rescue com.huaweicloud.dis.adapter.kafka.common.errors.SerializationException => e
186
+ # TODO(sissel): Retrying will fail because the data itself has a problem serializing.
187
+ # TODO(sissel): Let's add DLQ here.
188
+ failures << record
189
+ nil
190
+ end
191
+ end.compact
192
+
193
+ end
194
+
195
+ end
196
+
197
+ def close
198
+ @producer.close
199
+ end
200
+
201
+ private
202
+
203
+ def write_to_dis(event, serialized_data)
204
+ stream = event.get("stream");
205
+ if stream.nil?
206
+ stream = @stream;
207
+ end
208
+
209
+ message_key = event.get("partition_key");
210
+ if message_key.nil?
211
+ message_key = @message_key;
212
+ end
213
+
214
+ partition_id = event.get("partition_id");
215
+
216
+ if message_key.nil? && partition_id.nil?
217
+ # record = ProducerRecord.new(event.sprintf(@stream), serialized_data)
218
+ record = ProducerRecord.new(stream, serialized_data)
219
+ elsif partition_id.nil?
220
+ # record = ProducerRecord.new(event.sprintf(@stream), event.sprintf(@message_key), serialized_data)
221
+ # record = ProducerRecord.new(stream, event.sprintf(@message_key), serialized_data)
222
+ record = ProducerRecord.new(stream, message_key, serialized_data)
223
+ else
224
+ record = ProducerRecord.new(stream, partition_id.to_i, message_key, serialized_data)
225
+ end
226
+ prepare(record)
227
+ rescue LogStash::ShutdownSignal
228
+ @logger.debug('DIS Kafka producer got shutdown signal')
229
+ # sleep for 5 second to guaranteed data transmission is completed
230
+ sleep(5)
231
+ rescue => e
232
+ @logger.warn('DIS kafka producer threw exception, restarting',
233
+ :exception => e)
234
+ end
235
+
236
+ def create_producer
237
+ begin
238
+ props = java.util.Properties.new
239
+ kafka = com.huaweicloud.dis.adapter.kafka.clients.producer.ProducerConfig
240
+
241
+ props.put("IS_DEFAULT_TRUSTED_JKS_ENABLED", default_trusted_jks_enabled.to_s)
242
+ props.put("security.token", security_token) unless security_token.nil?
243
+ props.put("exception.retries", exception_retries.to_s)
244
+ props.put("records.retries", records_retries.to_s)
245
+ props.put("PROXY_HOST", proxy_host) unless proxy_host.nil?
246
+ props.put("PROXY_PORT", proxy_port.to_s)
247
+ props.put("PROXY_PROTOCOL", proxy_protocol)
248
+ props.put("PROXY_USERNAME", proxy_username) unless proxy_username.nil?
249
+ props.put("PROXY_PASSWORD", proxy_password) unless proxy_password.nil?
250
+ props.put("PROXY_WORKSTATION", proxy_workstation) unless proxy_workstation.nil?
251
+ props.put("PROXY_DOMAIN", proxy_domain) unless proxy_domain.nil?
252
+ props.put("NON_PROXY_HOSTS", proxy_non_proxy_hosts) unless proxy_non_proxy_hosts.nil?
253
+
254
+ props.put("batch.size", batch_size.to_s)
255
+ props.put("batch.count", batch_count.to_s)
256
+ props.put("buffer.memory", buffer_memory.to_s)
257
+ props.put("buffer.count", buffer_count.to_s)
258
+ props.put("linger.ms", linger_ms.to_s)
259
+ props.put("block.on.buffer.full", block_on_buffer_full.to_s)
260
+ props.put("max.block.ms", max_block_ms.to_s)
261
+ props.put("backoff.max.interval.ms", backoff_max_interval_ms.to_s)
262
+ props.put("max.in.flight.requests.per.connection", max_in_flight_requests_per_connection.to_s)
263
+ props.put("records.retriable.error.code", records_retriable_error_code) unless records_retriable_error_code.nil?
264
+ props.put("order.by.partition", order_by_partition.to_s)
265
+ props.put("body.serialize.type", body_serialize_type.to_s)
266
+ props.put("metadata.timeout.ms", metadata_timeout_ms.to_s)
267
+ # props.put(kafka::RETRIES_CONFIG, retries.to_s) unless retries.nil?
268
+ # props.put(kafka::RETRY_BACKOFF_MS_CONFIG, retry_backoff_ms.to_s)
269
+ props.put("key.deserializer", "com.huaweicloud.dis.adapter.kafka.common.serialization.StringDeserializer")
270
+ props.put("value.deserializer", "com.huaweicloud.dis.adapter.kafka.common.serialization.StringDeserializer")
271
+
272
+ # endpoint, project_id, region, ak, sk
273
+ props.put("endpoint", endpoint)
274
+ props.put("projectId", project_id)
275
+ props.put("region", region)
276
+ props.put("ak", ak)
277
+ if is_sk_encrypted
278
+ decrypted_sk = decrypt(@sk)
279
+ props.put("sk", decrypted_sk)
280
+ else
281
+ props.put("sk", sk)
282
+ end
283
+
284
+
285
+ com.huaweicloud.dis.adapter.kafka.clients.producer.DISKafkaProducer.new(props)
286
+ rescue => e
287
+ logger.error("Unable to create DIS Kafka producer from given configuration",
288
+ :kafka_error_message => e,
289
+ :cause => e.respond_to?(:getCause) ? e.getCause() : nil)
290
+ raise e
291
+ end
292
+ end
293
+
294
+ private
295
+ def decrypt(encrypted_sk)
296
+ com.huaweicloud.dis.util.encrypt.EncryptUtils.dec([@encrypt_key].to_java(java.lang.String), encrypted_sk)
297
+ rescue => e
298
+ logger.error("Unable to decrypt sk from given configuration",
299
+ :decrypt_error_message => e,
300
+ :cause => e.respond_to?(:getCause) ? e.getCause() : nil)
301
+ end
302
+
303
+ end #class LogStash::Outputs::Dis
@@ -1,5 +1,5 @@
1
- # encoding: utf-8
2
- require 'logstash/environment'
3
-
4
- root_dir = File.expand_path(File.join(File.dirname(__FILE__), ".."))
5
- LogStash::Environment.load_runtime_jars! File.join(root_dir, "vendor")
1
+ # encoding: utf-8
2
+ require 'logstash/environment'
3
+
4
+ root_dir = File.expand_path(File.join(File.dirname(__FILE__), ".."))
5
+ LogStash::Environment.load_runtime_jars! File.join(root_dir, "vendor")
@@ -1,36 +1,36 @@
1
- Gem::Specification.new do |s|
2
-
3
- s.name = 'logstash-output-dis'
4
- s.version = '1.1.2'
5
- s.licenses = ['Apache License (2.0)']
6
- s.summary = "Writes events to a DIS stream"
7
- s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
8
- s.authors = ['Data Ingestion Service']
9
- s.email = 'dis@huaweicloud.com'
10
- s.homepage = "https://www.huaweicloud.com/product/dis.html"
11
- s.require_paths = ['lib', 'vendor/jar-dependencies']
12
-
13
- # Files
14
- s.files = Dir["lib/**/*","spec/**/*","*.gemspec","*.md","CONTRIBUTORS","Gemfile","LICENSE","NOTICE.TXT", "vendor/jar-dependencies/**/*.jar", "vendor/jar-dependencies/**/*.rb", "VERSION", "docs/**/*"]
15
-
16
- # Tests
17
- s.test_files = s.files.grep(%r{^(test|spec|features)/})
18
-
19
- # Special flag to let us know this is actually a logstash plugin
20
- s.metadata = { 'logstash_plugin' => 'true', 'group' => 'output'}
21
-
22
- s.requirements << "jar 'com.huaweicloud.dis:huaweicloud-dis-kafka-adapter', '1.2.1'"
23
- s.requirements << "jar 'org.slf4j:slf4j-log4j12', '1.7.21'"
24
- s.requirements << "jar 'org.apache.logging.log4j:log4j-1.2-api', '2.6.2'"
25
-
26
- s.add_development_dependency 'jar-dependencies', '~> 0.3.2'
27
-
28
- # Gem dependencies
29
- s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
30
- s.add_runtime_dependency 'logstash-codec-plain'
31
- s.add_runtime_dependency 'logstash-codec-json'
32
-
33
- s.add_development_dependency 'logstash-devutils'
34
- s.add_development_dependency 'poseidon'
35
- s.add_development_dependency 'snappy'
36
- end
1
+ Gem::Specification.new do |s|
2
+
3
+ s.name = 'logstash-output-dis'
4
+ s.version = '1.1.3'
5
+ s.licenses = ['Apache License (2.0)']
6
+ s.summary = "Writes events to a DIS stream"
7
+ s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
8
+ s.authors = ['Data Ingestion Service']
9
+ s.email = 'dis@huaweicloud.com'
10
+ s.homepage = "https://www.huaweicloud.com/product/dis.html"
11
+ s.require_paths = ['lib', 'vendor/jar-dependencies']
12
+
13
+ # Files
14
+ s.files = Dir["lib/**/*","spec/**/*","*.gemspec","*.md","CONTRIBUTORS","Gemfile","LICENSE","NOTICE.TXT", "vendor/jar-dependencies/**/*.jar", "vendor/jar-dependencies/**/*.rb", "VERSION", "docs/**/*"]
15
+
16
+ # Tests
17
+ s.test_files = s.files.grep(%r{^(test|spec|features)/})
18
+
19
+ # Special flag to let us know this is actually a logstash plugin
20
+ s.metadata = { 'logstash_plugin' => 'true', 'group' => 'output'}
21
+
22
+ s.requirements << "jar 'com.huaweicloud.dis:huaweicloud-dis-kafka-adapter', '1.2.1'"
23
+ s.requirements << "jar 'org.slf4j:slf4j-log4j12', '1.7.21'"
24
+ s.requirements << "jar 'org.apache.logging.log4j:log4j-1.2-api', '2.6.2'"
25
+
26
+ s.add_development_dependency 'jar-dependencies', '~> 0.3.2'
27
+
28
+ # Gem dependencies
29
+ s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
30
+ s.add_runtime_dependency 'logstash-codec-plain'
31
+ s.add_runtime_dependency 'logstash-codec-json'
32
+
33
+ s.add_development_dependency 'logstash-devutils'
34
+ s.add_development_dependency 'poseidon'
35
+ s.add_development_dependency 'snappy'
36
+ end
@@ -1,128 +1,128 @@
1
- # encoding: utf-8
2
- require "logstash/devutils/rspec/spec_helper"
3
- require 'logstash/outputs/dis'
4
- require 'json'
5
-
6
- describe "outputs/dis" do
7
- let (:simple_dis_config) {{'stream' => 'test', 'project_id' => 'test_project_id', 'ak' => 'test_ak', 'sk' => 'test_sk'}}
8
- let (:event) { LogStash::Event.new({'message' => 'hello', 'stream_name' => 'my_stream', 'host' => '127.0.0.1',
9
- '@timestamp' => LogStash::Timestamp.now}) }
10
-
11
- context 'when initializing' do
12
- it "should register" do
13
- output = LogStash::Plugin.lookup("output", "dis").new(simple_dis_config)
14
- expect {output.register}.to_not raise_error
15
- end
16
-
17
- it 'should populate dis config with default values' do
18
- dis = LogStash::Outputs::Dis.new(simple_dis_config)
19
- insist {dis.endpoint} == 'https://dis.cn-north-1.myhuaweicloud.com'
20
- insist {dis.stream} == 'test'
21
- insist {dis.key_serializer} == 'com.huaweicloud.dis.adapter.kafka.common.serialization.StringSerializer'
22
- end
23
- end
24
-
25
- context 'when outputting messages' do
26
- #it 'should send logstash event to DIS' do
27
- #expect_any_instance_of(com.huaweicloud.dis.adapter.kafka.clients.producer.DISKafkaProducer).to receive(:send)
28
- #.with(an_instance_of(com.huaweicloud.dis.adapter.kafka.clients.producer.ProducerRecord)).and_call_original
29
- #dis = LogStash::Outputs::Dis.new(simple_dis_config)
30
- #dis.register
31
- #dis.multi_receive([event])
32
- #end
33
-
34
- #it 'should support field referenced message_keys' do
35
- #expect(com.huaweicloud.dis.adapter.kafka.clients.producer.ProducerRecord).to receive(:new)
36
- #.with("test", "127.0.0.1", event.to_s).and_call_original
37
- #expect_any_instance_of(com.huaweicloud.dis.adapter.kafka.clients.producer.DISKafkaProducer).to receive(:send).and_call_original
38
- #dis = LogStash::Outputs::Dis.new(simple_dis_config.merge({"message_key" => "%{host}"}))
39
- #dis.register
40
- #dis.multi_receive([event])
41
- #end
42
- end
43
-
44
- context "when DISKafkaProducer#send() raises an exception" do
45
- let(:failcount) { (rand * 10).to_i }
46
- let(:sendcount) { failcount + 1 }
47
-
48
- let(:exception_classes) { [
49
- com.huaweicloud.dis.adapter.kafka.common.errors.TimeoutException,
50
- com.huaweicloud.dis.adapter.kafka.common.errors.InterruptException,
51
- com.huaweicloud.dis.adapter.kafka.common.errors.SerializationException
52
- ] }
53
-
54
- before do
55
- count = 0
56
- expect_any_instance_of(com.huaweicloud.dis.adapter.kafka.clients.producer.DISKafkaProducer).to receive(:send)
57
- .exactly(sendcount).times
58
- .and_wrap_original do |m, *args|
59
- if count < failcount # fail 'failcount' times in a row.
60
- count += 1
61
- # Pick an exception at random
62
- raise exception_classes.shuffle.first.new("injected exception for testing")
63
- else
64
- #m.call(*args) # call original
65
- end
66
- end
67
- end
68
-
69
- it "should retry until successful" do
70
- dis = LogStash::Outputs::Dis.new(simple_dis_config)
71
- dis.register
72
- dis.multi_receive([event])
73
- end
74
- end
75
-
76
- context "when a send fails" do
77
- context "and the default retries behavior is used" do
78
- # Fail this many times and then finally succeed.
79
- let(:failcount) { (rand * 10).to_i }
80
-
81
- # Expect DISKafkaProducer.send() to get called again after every failure, plus the successful one.
82
- let(:sendcount) { failcount + 1 }
83
-
84
-
85
- it "should retry until successful" do
86
- count = 0;
87
-
88
- expect_any_instance_of(com.huaweicloud.dis.adapter.kafka.clients.producer.DISKafkaProducer).to receive(:send)
89
- .exactly(sendcount).times
90
- .and_wrap_original do |m, *args|
91
- if count < failcount
92
- count += 1
93
- # inject some failures.
94
-
95
- # Return a custom Future that will raise an exception to simulate a DIS send() problem.
96
- future = java.util.concurrent.FutureTask.new { raise "Failed" }
97
- future.run
98
- future
99
- else
100
- #m.call(*args)
101
- end
102
- end
103
- dis = LogStash::Outputs::Dis.new(simple_dis_config)
104
- dis.register
105
- dis.multi_receive([event])
106
- end
107
- end
108
-
109
- context "and when retries is set by the user" do
110
- let(:retries) { (rand * 10).to_i }
111
- let(:max_sends) { retries + 1 }
112
-
113
- it "should give up after retries are exhausted" do
114
- expect_any_instance_of(com.huaweicloud.dis.adapter.kafka.clients.producer.DISKafkaProducer).to receive(:send)
115
- .at_most(max_sends).times
116
- .and_wrap_original do |m, *args|
117
- # Always fail.
118
- future = java.util.concurrent.FutureTask.new { raise "Failed" }
119
- future.run
120
- future
121
- end
122
- dis = LogStash::Outputs::Dis.new(simple_dis_config.merge("retries" => retries))
123
- dis.register
124
- dis.multi_receive([event])
125
- end
126
- end
127
- end
128
- end
1
+ # encoding: utf-8
2
+ require "logstash/devutils/rspec/spec_helper"
3
+ require 'logstash/outputs/dis'
4
+ require 'json'
5
+
6
+ describe "outputs/dis" do
7
+ let (:simple_dis_config) {{'stream' => 'test', 'project_id' => 'test_project_id', 'ak' => 'test_ak', 'sk' => 'test_sk'}}
8
+ let (:event) { LogStash::Event.new({'message' => 'hello', 'stream_name' => 'my_stream', 'host' => '127.0.0.1',
9
+ '@timestamp' => LogStash::Timestamp.now}) }
10
+
11
+ context 'when initializing' do
12
+ it "should register" do
13
+ output = LogStash::Plugin.lookup("output", "dis").new(simple_dis_config)
14
+ expect {output.register}.to_not raise_error
15
+ end
16
+
17
+ it 'should populate dis config with default values' do
18
+ dis = LogStash::Outputs::Dis.new(simple_dis_config)
19
+ insist {dis.endpoint} == 'https://dis.cn-north-1.myhuaweicloud.com'
20
+ insist {dis.stream} == 'test'
21
+ insist {dis.key_serializer} == 'com.huaweicloud.dis.adapter.kafka.common.serialization.StringSerializer'
22
+ end
23
+ end
24
+
25
+ context 'when outputting messages' do
26
+ #it 'should send logstash event to DIS' do
27
+ #expect_any_instance_of(com.huaweicloud.dis.adapter.kafka.clients.producer.DISKafkaProducer).to receive(:send)
28
+ #.with(an_instance_of(com.huaweicloud.dis.adapter.kafka.clients.producer.ProducerRecord)).and_call_original
29
+ #dis = LogStash::Outputs::Dis.new(simple_dis_config)
30
+ #dis.register
31
+ #dis.multi_receive([event])
32
+ #end
33
+
34
+ #it 'should support field referenced message_keys' do
35
+ #expect(com.huaweicloud.dis.adapter.kafka.clients.producer.ProducerRecord).to receive(:new)
36
+ #.with("test", "127.0.0.1", event.to_s).and_call_original
37
+ #expect_any_instance_of(com.huaweicloud.dis.adapter.kafka.clients.producer.DISKafkaProducer).to receive(:send).and_call_original
38
+ #dis = LogStash::Outputs::Dis.new(simple_dis_config.merge({"message_key" => "%{host}"}))
39
+ #dis.register
40
+ #dis.multi_receive([event])
41
+ #end
42
+ end
43
+
44
+ context "when DISKafkaProducer#send() raises an exception" do
45
+ let(:failcount) { (rand * 10).to_i }
46
+ let(:sendcount) { failcount + 1 }
47
+
48
+ let(:exception_classes) { [
49
+ com.huaweicloud.dis.adapter.kafka.common.errors.TimeoutException,
50
+ com.huaweicloud.dis.adapter.kafka.common.errors.InterruptException,
51
+ com.huaweicloud.dis.adapter.kafka.common.errors.SerializationException
52
+ ] }
53
+
54
+ before do
55
+ count = 0
56
+ expect_any_instance_of(com.huaweicloud.dis.adapter.kafka.clients.producer.DISKafkaProducer).to receive(:send)
57
+ .exactly(sendcount).times
58
+ .and_wrap_original do |m, *args|
59
+ if count < failcount # fail 'failcount' times in a row.
60
+ count += 1
61
+ # Pick an exception at random
62
+ raise exception_classes.shuffle.first.new("injected exception for testing")
63
+ else
64
+ #m.call(*args) # call original
65
+ end
66
+ end
67
+ end
68
+
69
+ it "should retry until successful" do
70
+ dis = LogStash::Outputs::Dis.new(simple_dis_config)
71
+ dis.register
72
+ dis.multi_receive([event])
73
+ end
74
+ end
75
+
76
+ context "when a send fails" do
77
+ context "and the default retries behavior is used" do
78
+ # Fail this many times and then finally succeed.
79
+ let(:failcount) { (rand * 10).to_i }
80
+
81
+ # Expect DISKafkaProducer.send() to get called again after every failure, plus the successful one.
82
+ let(:sendcount) { failcount + 1 }
83
+
84
+
85
+ it "should retry until successful" do
86
+ count = 0;
87
+
88
+ expect_any_instance_of(com.huaweicloud.dis.adapter.kafka.clients.producer.DISKafkaProducer).to receive(:send)
89
+ .exactly(sendcount).times
90
+ .and_wrap_original do |m, *args|
91
+ if count < failcount
92
+ count += 1
93
+ # inject some failures.
94
+
95
+ # Return a custom Future that will raise an exception to simulate a DIS send() problem.
96
+ future = java.util.concurrent.FutureTask.new { raise "Failed" }
97
+ future.run
98
+ future
99
+ else
100
+ #m.call(*args)
101
+ end
102
+ end
103
+ dis = LogStash::Outputs::Dis.new(simple_dis_config)
104
+ dis.register
105
+ dis.multi_receive([event])
106
+ end
107
+ end
108
+
109
+ context "and when retries is set by the user" do
110
+ let(:retries) { (rand * 10).to_i }
111
+ let(:max_sends) { retries + 1 }
112
+
113
+ it "should give up after retries are exhausted" do
114
+ expect_any_instance_of(com.huaweicloud.dis.adapter.kafka.clients.producer.DISKafkaProducer).to receive(:send)
115
+ .at_most(max_sends).times
116
+ .and_wrap_original do |m, *args|
117
+ # Always fail.
118
+ future = java.util.concurrent.FutureTask.new { raise "Failed" }
119
+ future.run
120
+ future
121
+ end
122
+ dis = LogStash::Outputs::Dis.new(simple_dis_config.merge("retries" => retries))
123
+ dis.register
124
+ dis.multi_receive([event])
125
+ end
126
+ end
127
+ end
128
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-output-dis
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.1.2
4
+ version: 1.1.3
5
5
  platform: ruby
6
6
  authors:
7
7
  - Data Ingestion Service
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2019-04-24 00:00:00.000000000 Z
11
+ date: 2019-07-07 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  requirement: !ruby/object:Gem::Requirement