logstash-output-s3 4.3.5 → 4.3.6

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: bc8cc704a468f0576b34c9fad38dd2506ea303c4b8671e8f575a48c7e1f4e602
4
- data.tar.gz: d39aaebe320c7a3dd109a5b35b3e5eaf9b06c9f382b88ca934e51c4ae990491d
3
+ metadata.gz: 4671860be7ad0735c12ef8f886c157dce2990975a30022a07b9f95cfe5187e3b
4
+ data.tar.gz: e17d8f575abcd12968b5a4065be4786f904536aa0e77a2c48854e20ac259ddf3
5
5
  SHA512:
6
- metadata.gz: 67c47cbdcf47983beed7edd157541f5d614c300c6c28f2eccd9ed039a1cd1b29eeab4f512c7b45e3a8f2936a128e95ca028f4b4b94797c8217bd9144bad6e6ab
7
- data.tar.gz: faeeb2f29ac962ebc50dc2a874617f85f3d7dbb882ab754b7a6fc571c721d416ddb25e63ce01ab951b18b9585e3b287faadea612b99790dee624ec82c61303a4
6
+ metadata.gz: 3c20213ce22c85afc5512adb8641aa72c365f0644b42939acd40fb189c846118153523b8de749bb8b19e866faed7c80e870494d61c4f99b1691417f0618fcc8e
7
+ data.tar.gz: 046f0d0f0ceaf6cc69c3494348d0e6b867c919be4d1a053b3c1fcbe0fe5591e47a97fd7e71b2606fa0949a72fc807bf1877a73cb4cbdd30bf091d91317c7a261
data/CHANGELOG.md CHANGED
@@ -1,20 +1,24 @@
1
+ ## 4.3.6
2
+ - Docs: more documentation on restore + temp dir [#236](https://github.com/logstash-plugins/logstash-output-s3/pull/236)
3
+ * minor logging improvements - use the same path: naming convention
4
+
1
5
  ## 4.3.5
2
- - Feat: cast true/false values for additional_settings [#241](https://github.com/logstash-plugins/logstash-output-s3/pull/241)
6
+ - Feat: cast true/false values for additional_settings [#241](https://github.com/logstash-plugins/logstash-output-s3/pull/241)
3
7
 
4
8
  ## 4.3.4
5
- - [DOC] Added note about performance implications of interpolated strings in prefixes [#233](https://github.com/logstash-plugins/logstash-output-s3/pull/233)
9
+ - [DOC] Added note about performance implications of interpolated strings in prefixes [#233](https://github.com/logstash-plugins/logstash-output-s3/pull/233)
6
10
 
7
11
  ## 4.3.3
8
- - [DOC] Updated links to use shared attributes [#230](https://github.com/logstash-plugins/logstash-output-s3/pull/230)
12
+ - [DOC] Updated links to use shared attributes [#230](https://github.com/logstash-plugins/logstash-output-s3/pull/230)
9
13
 
10
14
  ## 4.3.2
11
- - [DOC] Added note that only AWS S3 is supported. No other S3 compatible storage solutions are supported. [#223](https://github.com/logstash-plugins/logstash-output-s3/pull/223)
15
+ - [DOC] Added note that only AWS S3 is supported. No other S3 compatible storage solutions are supported. [#223](https://github.com/logstash-plugins/logstash-output-s3/pull/223)
12
16
 
13
17
  ## 4.3.1
14
- - [DOC] Updated setting descriptions for clarity [#219](https://github.com/logstash-plugins/logstash-output-s3/pull/219) and [#220](https://github.com/logstash-plugins/logstash-output-s3/pull/220)
18
+ - [DOC] Updated setting descriptions for clarity [#219](https://github.com/logstash-plugins/logstash-output-s3/pull/219) and [#220](https://github.com/logstash-plugins/logstash-output-s3/pull/220)
15
19
 
16
20
  ## 4.3.0
17
- - Feat: Added retry_count and retry_delay config [#218](https://github.com/logstash-plugins/logstash-output-s3/pull/218)
21
+ - Feat: Added retry_count and retry_delay config [#218](https://github.com/logstash-plugins/logstash-output-s3/pull/218)
18
22
 
19
23
  ## 4.2.0
20
24
  - Added ability to specify [ONEZONE_IA](https://aws.amazon.com/s3/storage-classes/#__) as storage_class
data/docs/index.asciidoc CHANGED
@@ -30,8 +30,9 @@ Other S3 compatible storage solutions are not supported.
30
30
  S3 outputs create temporary files into the OS' temporary directory.
31
31
  You can specify where to save them using the `temporary_directory` option.
32
32
 
33
- IMPORTANT: For configurations containing multiple s3 outputs with the restore
34
- option enabled, each output should define its own 'temporary_directory'.
33
+ IMPORTANT: For configurations containing multiple s3 outputs with the `restore`
34
+ option enabled, each output should define its own `temporary_directory`.
35
+ Shared or nested directories can cause data loss upon recovery.
35
36
 
36
37
  ===== Requirements
37
38
 
@@ -255,6 +256,10 @@ The AWS Region
255
256
  Used to enable recovery after crash/abnormal termination.
256
257
  Temporary log files will be recovered and uploaded.
257
258
 
259
+ NOTE: If you're using multiple S3 outputs, always set
260
+ <<plugins-{type}s-{plugin}-temporary_directory>> to a
261
+ unique directory. Otherwise the recovery mechanism won't work correctly.
262
+
258
263
  [id="plugins-{type}s-{plugin}-retry_count"]
259
264
  ===== `retry_count`
260
265
 
@@ -388,7 +393,12 @@ Defaults to STANDARD.
388
393
  * Default value is `"/tmp/logstash"`
389
394
 
390
395
  Set the directory where logstash will store the tmp files before sending it to S3
391
- default to the current OS temporary directory in linux /tmp/logstash
396
+ default to the current OS temporary directory in linux `/tmp/logstash`.
397
+
398
+ WARNING: Using multiple S3 outputs with `restore => true` requires unique directories
399
+ per output. All of the directory's contents are processed and deleted upon recovery, and shared or nested directories can cause data loss.
400
+ For example, an output using `/tmp/s3` and a second configured with `/tmp/s3/sub` would
401
+ cause issues. Having temporary directories `/tmp/s3/sub1` and `/tmp/s3/sub2` is fine.
392
402
 
393
403
  [id="plugins-{type}s-{plugin}-time_file"]
394
404
  ===== `time_file`
@@ -4,8 +4,6 @@ require "concurrent"
4
4
  require "concurrent/timer_task"
5
5
  require "logstash/util"
6
6
 
7
- ConcurrentHashMap = java.util.concurrent.ConcurrentHashMap
8
-
9
7
  module LogStash
10
8
  module Outputs
11
9
  class S3
@@ -59,7 +57,7 @@ module LogStash
59
57
  sweeper_interval = DEFAULT_STATE_SWEEPER_INTERVAL_SECS)
60
58
  # The path need to contains the prefix so when we start
61
59
  # logtash after a crash we keep the remote structure
62
- @prefixed_factories = ConcurrentHashMap.new
60
+ @prefixed_factories = java.util.concurrent.ConcurrentHashMap.new
63
61
 
64
62
  @sweeper_interval = sweeper_interval
65
63
 
@@ -110,7 +110,8 @@ class LogStash::Outputs::S3 < LogStash::Outputs::Base
110
110
 
111
111
  # Set the size of file in bytes, this means that files on bucket when have dimension > file_size, they are stored in two or more file.
112
112
  # If you have tags then it will generate a specific size file for every tags
113
- ##NOTE: define size of file is the better thing, because generate a local temporary file on disk and then put it in bucket.
113
+ #
114
+ # NOTE: define size of file is the better thing, because generate a local temporary file on disk and then put it in bucket.
114
115
  config :size_file, :validate => :number, :default => 1024 * 1024 * 5
115
116
 
116
117
  # Set the time, in MINUTES, to close the current sub_time_section of bucket.
@@ -118,10 +119,10 @@ class LogStash::Outputs::S3 < LogStash::Outputs::Base
118
119
  # If it's valued 0 and rotation_strategy is 'time' or 'size_and_time' then the plugin reaise a configuration error.
119
120
  config :time_file, :validate => :number, :default => 15
120
121
 
121
- ## IMPORTANT: if you use multiple instance of s3, you should specify on one of them the "restore=> true" and on the others "restore => false".
122
- ## This is hack for not destroy the new files after restoring the initial files.
123
- ## If you do not specify "restore => true" when logstash crashes or is restarted, the files are not sent into the bucket,
124
- ## for example if you have single Instance.
122
+ # If `restore => false` is specified and Logstash crashes, the unprocessed files are not sent into the bucket.
123
+ #
124
+ # NOTE: that the `recovery => true` default assumes multiple S3 outputs would set a unique `temporary_directory => ...`
125
+ # if they do not than only a single S3 output is safe to recover (since let-over files are processed and deleted).
125
126
  config :restore, :validate => :boolean, :default => true
126
127
 
127
128
  # The S3 canned ACL to use when putting the file. Defaults to "private".
@@ -147,6 +148,9 @@ class LogStash::Outputs::S3 < LogStash::Outputs::Base
147
148
 
148
149
  # Set the directory where logstash will store the tmp files before sending it to S3
149
150
  # default to the current OS temporary directory in linux /tmp/logstash
151
+ #
152
+ # NOTE: the reason we do not have a unique (isolated) temporary directory as a default, to support multiple plugin instances,
153
+ # is that we would have to rely on something static that does not change between restarts (e.g. a user set id => ...).
150
154
  config :temporary_directory, :validate => :string, :default => File.join(Dir.tmpdir, "logstash")
151
155
 
152
156
  # Specify a prefix to the uploaded filename, this can simulate directories on S3. Prefix does not require leading slash.
@@ -347,10 +351,10 @@ class LogStash::Outputs::S3 < LogStash::Outputs::Base
347
351
  temp_file = factory.current
348
352
 
349
353
  if @rotation.rotate?(temp_file)
350
- @logger.debug("Rotate file",
351
- :strategy => @rotation.class.name,
352
- :key => temp_file.key,
353
- :path => temp_file.path)
354
+ @logger.debug? && @logger.debug("Rotate file",
355
+ :key => temp_file.key,
356
+ :path => temp_file.path,
357
+ :strategy => @rotation.class.name)
354
358
 
355
359
  upload_file(temp_file)
356
360
  factory.rotate!
@@ -360,7 +364,7 @@ class LogStash::Outputs::S3 < LogStash::Outputs::Base
360
364
  end
361
365
 
362
366
  def upload_file(temp_file)
363
- @logger.debug("Queue for upload", :path => temp_file.path)
367
+ @logger.debug? && @logger.debug("Queue for upload", :path => temp_file.path)
364
368
 
365
369
  # if the queue is full the calling thread will be used to upload
366
370
  temp_file.close # make sure the content is on disk
@@ -383,7 +387,7 @@ class LogStash::Outputs::S3 < LogStash::Outputs::Base
383
387
  end
384
388
 
385
389
  def clean_temporary_file(file)
386
- @logger.debug("Removing temporary file", :file => file.path)
390
+ @logger.debug? && @logger.debug("Removing temporary file", :path => file.path)
387
391
  file.delete!
388
392
  end
389
393
 
@@ -398,7 +402,7 @@ class LogStash::Outputs::S3 < LogStash::Outputs::Base
398
402
  .each do |file|
399
403
  temp_file = TemporaryFile.create_from_existing_file(file, temp_folder_path)
400
404
  if temp_file.size > 0
401
- @logger.debug("Recovering from crash and uploading", :file => temp_file.path)
405
+ @logger.debug? && @logger.debug("Recovering from crash and uploading", :path => temp_file.path)
402
406
  @crash_uploader.upload_async(temp_file, :on_complete => method(:clean_temporary_file), :upload_options => upload_options)
403
407
  else
404
408
  clean_temporary_file(temp_file)
@@ -1,6 +1,6 @@
1
1
  Gem::Specification.new do |s|
2
2
  s.name = 'logstash-output-s3'
3
- s.version = '4.3.5'
3
+ s.version = '4.3.6'
4
4
  s.licenses = ['Apache-2.0']
5
5
  s.summary = "Sends Logstash events to the Amazon Simple Storage Service"
6
6
  s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
@@ -88,8 +88,8 @@ describe LogStash::Outputs::S3::FileRepository do
88
88
 
89
89
  it "returns all available keys" do
90
90
  subject.get_file(prefix_key) { |file| file.write("something") }
91
- expect(subject.keys.toArray).to include(prefix_key)
92
- expect(subject.keys.toArray.size).to eq(1)
91
+ expect(subject.keys).to include(prefix_key)
92
+ expect(subject.keys.to_a.size).to eq(1)
93
93
  end
94
94
 
95
95
  it "clean stale factories" do
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-output-s3
3
3
  version: !ruby/object:Gem::Version
4
- version: 4.3.5
4
+ version: 4.3.6
5
5
  platform: ruby
6
6
  authors:
7
7
  - Elastic
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2021-10-06 00:00:00.000000000 Z
11
+ date: 2022-06-14 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  requirement: !ruby/object:Gem::Requirement