logstash-input-s3-sns-sqs 2.2.3.pre → 2.2.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/AGENTS.md +217 -0
- data/CHANGELOG.md +8 -2
- data/lib/logstash/inputs/s3/downloader.rb +1 -1
- data/lib/logstash/inputs/s3snssqs.rb +15 -62
- data/lib/logstash/plugin_mixins/aws_config/v2.rb +1 -1
- data/logstash-input-s3-sns-sqs.gemspec +13 -2
- metadata +39 -41
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 6fe9a8dad0beb85243fa668010ae55535fd37ad7888befbcfb24894446023dc1
|
|
4
|
+
data.tar.gz: cc2176146a6453abb31c0829acab94fe72fb1e12a8c4e0c6b0547dabee8871bf
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 9b31eafd90e6d6c9d1ce22bab16ecdca5e6e9bd84eed8971a98e718c0634692d679d8870b8eefa503c2572f4ca08caed77249a123a3e80d4a55b6e5f36e46e56
|
|
7
|
+
data.tar.gz: 3d63bda499713dc2d5267c4abf3a63b62ddfbfcdd39e32eb34b6883bfa30532795e30c0ed3e8bd55df00de05b12d9db2e94aeeca64e200160de8cfaaf2afb256
|
data/AGENTS.md
ADDED
|
@@ -0,0 +1,217 @@
|
|
|
1
|
+
# AGENTS.md — logstash-input-s3-sns-sqs
|
|
2
|
+
|
|
3
|
+
> Project-specific guidance for AI agents working in this repo.
|
|
4
|
+
> **Read this before making any code / tooling / release changes.**
|
|
5
|
+
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## 1. What this project is
|
|
9
|
+
|
|
10
|
+
A **Logstash input plugin** written in Ruby (runs on JRuby, since Logstash uses JRuby).
|
|
11
|
+
It ingests S3 object-created events via SQS (optionally via SNS → SQS fan-out), downloads
|
|
12
|
+
the referenced S3 objects, and emits Logstash events from their contents.
|
|
13
|
+
|
|
14
|
+
- Gem name: `logstash-input-s3-sns-sqs`
|
|
15
|
+
- Current version: `2.2.2` (see `logstash-input-s3-sns-sqs.gemspec`)
|
|
16
|
+
- Author: Christian Herweg
|
|
17
|
+
- License: Apache-2.0
|
|
18
|
+
- Target: Logstash 8 / 9. The current Logstash 9.4.4 image ships
|
|
19
|
+
**JRuby 10.0.6 running Ruby 3.4.5** (verified against
|
|
20
|
+
`docker.elastic.co/logstash/logstash:9.4.4`), so any Ruby-3.2+ removal
|
|
21
|
+
(`File.exists?`, `Fixnum`, `TRUE`/`FALSE`, hash-as-kwarg) will crash there.
|
|
22
|
+
The gem itself still builds fine in `jruby:9.4` (Ruby 3.1) — see §3.1.
|
|
23
|
+
|
|
24
|
+
Key runtime dependencies (see gemspec):
|
|
25
|
+
`logstash-core-plugin-api ~> 2.0`, `aws-sdk-core ~> 3`, `aws-sdk-s3`, `aws-sdk-sqs`,
|
|
26
|
+
`aws-sdk-sns`, `concurrent-ruby`, `stud`, `logstash-codec-json`, `logstash-codec-plain`.
|
|
27
|
+
|
|
28
|
+
The plugin **includes an in-house `aws_config` mixin** under
|
|
29
|
+
`lib/logstash/plugin_mixins/aws_config/` — this was inlined to remove the
|
|
30
|
+
external `logstash-mixin-aws` dependency (see CHANGELOG 2.2.2: “Inhouse mixin components”).
|
|
31
|
+
Do not re-introduce that gem without discussing with maintainer.
|
|
32
|
+
|
|
33
|
+
---
|
|
34
|
+
|
|
35
|
+
## 2. Source layout
|
|
36
|
+
|
|
37
|
+
```
|
|
38
|
+
lib/logstash/
|
|
39
|
+
├── inputs/
|
|
40
|
+
│ ├── s3snssqs.rb # main input class LogStash::Inputs::S3SNSSQS
|
|
41
|
+
│ ├── codec_factory.rb # picks codec by folder/content-type
|
|
42
|
+
│ ├── mime/magic_gzip_validator.rb # small magic-byte gzip detection (perf)
|
|
43
|
+
│ ├── s3/
|
|
44
|
+
│ │ ├── client_factory.rb # per-bucket / per-region S3 client cache
|
|
45
|
+
│ │ └── downloader.rb # S3 GET + local temp file handling
|
|
46
|
+
│ ├── s3snssqs/
|
|
47
|
+
│ │ └── log_processor.rb # decompress + line-split + codec + emit event
|
|
48
|
+
│ └── sqs/
|
|
49
|
+
│ └── poller.rb # SQS receive loop, visibility, delete
|
|
50
|
+
└── plugin_mixins/
|
|
51
|
+
└── aws_config/
|
|
52
|
+
├── generic.rb # config DSL used by v2.rb
|
|
53
|
+
├── v2.rb # AWS SDK v2/v3 wiring: aws_options_hash
|
|
54
|
+
└── ...
|
|
55
|
+
spec/inputs/s3sqs_spec.rb # rspec (currently minimal; mostly disabled)
|
|
56
|
+
fixtures/log-stream.real-formatted # real sample used by the spec
|
|
57
|
+
docs/index.asciidoc # user-facing plugin documentation
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
### Threading model
|
|
61
|
+
- One SQS poller loop feeds a work queue.
|
|
62
|
+
- N `consumer_threads` decode S3 objects concurrently.
|
|
63
|
+
- A watcher thread restarts consumer threads that died with an exception
|
|
64
|
+
(feature added in 2.1.0).
|
|
65
|
+
|
|
66
|
+
### Message flow
|
|
67
|
+
1. `SQS::Poller#run` receives messages.
|
|
68
|
+
2. Message body is parsed as either a bare S3 event or an SNS-wrapped S3 event.
|
|
69
|
+
3. For each S3 record: `S3::Downloader#copy_s3object_to_disk` writes to
|
|
70
|
+
`temporary_directory`.
|
|
71
|
+
4. `S3SNSSQS::LogProcessor#process` reads (gunzip if needed), splits on `\n`,
|
|
72
|
+
pushes lines into the configured codec, decorates the event with S3 metadata,
|
|
73
|
+
and emits.
|
|
74
|
+
5. On success SQS message is deleted; on failure visibility timeout is used to
|
|
75
|
+
requeue.
|
|
76
|
+
|
|
77
|
+
---
|
|
78
|
+
|
|
79
|
+
## 3. How to build & test — **the only supported way locally**
|
|
80
|
+
|
|
81
|
+
**⚠️ You need JRuby.** MRI Ruby (`/usr/bin/ruby`) will *not* work — the plugin
|
|
82
|
+
`require`s `java_import` and depends on `logstash-core-plugin-api` which only
|
|
83
|
+
resolves on JRuby.
|
|
84
|
+
|
|
85
|
+
The maintainer does **not** ship a Dockerfile or CI workflow (no `.github/`,
|
|
86
|
+
no `.travis.yml`). Use the official `jruby:9.4` image via `podman` (or `docker`).
|
|
87
|
+
|
|
88
|
+
### 3.1 Quick recipe (podman on macOS)
|
|
89
|
+
|
|
90
|
+
```sh
|
|
91
|
+
# from repo root
|
|
92
|
+
podman run --rm -it \
|
|
93
|
+
-v "$PWD":/plugin -w /plugin \
|
|
94
|
+
docker.io/library/jruby:9.4 \
|
|
95
|
+
bash -lc "gem install bundler -v '~> 2.4' --no-doc && bundle install && bundle exec rspec --format documentation"
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
The first run downloads the JRuby image (~500 MB) and resolves ~200 gems.
|
|
99
|
+
Expect **5–15 minutes** on a cold cache. Subsequent runs with a mounted
|
|
100
|
+
`vendor/bundle` are seconds.
|
|
101
|
+
|
|
102
|
+
### 3.2 Cached run (recommended)
|
|
103
|
+
|
|
104
|
+
```sh
|
|
105
|
+
podman run --rm -it \
|
|
106
|
+
-v "$PWD":/plugin -w /plugin \
|
|
107
|
+
-e BUNDLE_PATH=/plugin/vendor/bundle \
|
|
108
|
+
docker.io/library/jruby:9.4 \
|
|
109
|
+
bash -lc "gem install bundler -v '~> 2.4' --no-doc && bundle install && bundle exec rspec"
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
Add `vendor/` to `.gitignore` (already ignored via `.vendor/`; consider `vendor/`).
|
|
113
|
+
|
|
114
|
+
### 3.3 What the current spec covers
|
|
115
|
+
|
|
116
|
+
`spec/inputs/s3sqs_spec.rb` is a smoke test. It only verifies the plugin loads
|
|
117
|
+
and `process` returns `true` on a real gzipped fixture. Most assertions are
|
|
118
|
+
commented out (see CHANGELOG 2.1.2 “CLEANUP: Remove tests”). **Do not treat
|
|
119
|
+
a green rspec as full validation.** Non-trivial changes should also be
|
|
120
|
+
smoke-tested inside a real Logstash pipeline.
|
|
121
|
+
|
|
122
|
+
### 3.4 Smoke test inside a real Logstash container
|
|
123
|
+
|
|
124
|
+
```sh
|
|
125
|
+
# Build the gem
|
|
126
|
+
podman run --rm -v "$PWD":/plugin -w /plugin docker.io/library/jruby:9.4 \
|
|
127
|
+
gem build logstash-input-s3-sns-sqs.gemspec
|
|
128
|
+
|
|
129
|
+
# Install into official Logstash and start with minimal config.
|
|
130
|
+
# Notes:
|
|
131
|
+
# --entrypoint "" -> image WORKDIR is /home, override needed
|
|
132
|
+
# --local -> skip artifacts.elastic.co pack lookup
|
|
133
|
+
# SSL_CERT_FILE -> needed behind corporate TLS MITM (see plugin-dev skill §0.1)
|
|
134
|
+
podman run --rm -v "$PWD":/plugin -w /usr/share/logstash --entrypoint "" \
|
|
135
|
+
-e SSL_CERT_FILE=/plugin/system-cas.pem \
|
|
136
|
+
docker.elastic.co/logstash/logstash:9.4.4 \
|
|
137
|
+
bash -lc "bin/logstash-plugin install --local /plugin/logstash-input-s3-sns-sqs-*.gem && bin/logstash -e 'input { s3snssqs { queue => \"dummy\" region => \"eu-central-1\" } } output { stdout {} }' --config.test_and_exit"
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
`--config.test_and_exit` validates that the plugin registers cleanly without
|
|
141
|
+
actually connecting to AWS.
|
|
142
|
+
|
|
143
|
+
---
|
|
144
|
+
|
|
145
|
+
## 4. Release process
|
|
146
|
+
|
|
147
|
+
Source of truth: `release.howto` (dated; commands below are the modernized
|
|
148
|
+
equivalent).
|
|
149
|
+
|
|
150
|
+
1. Bump `s.version` in `logstash-input-s3-sns-sqs.gemspec`.
|
|
151
|
+
2. Prepend a stanza to `CHANGELOG.md`. Follow the existing terse style
|
|
152
|
+
(`##<version>` header, `-` bullets, feature/fix/security tags).
|
|
153
|
+
3. Commit with message `prepare release <version>`.
|
|
154
|
+
4. Build & push:
|
|
155
|
+
```sh
|
|
156
|
+
# inside a jruby container with ~/.gem/credentials mounted
|
|
157
|
+
podman run --rm -it -v "$PWD":/plugin -v "$HOME/.gem":/root/.gem -w /plugin \
|
|
158
|
+
docker.io/library/jruby:9.4 \
|
|
159
|
+
bash -lc "bundle install && bundle exec rake vendor && bundle exec rake publish_gem --trace"
|
|
160
|
+
```
|
|
161
|
+
5. Tag the release: `git tag v<version> && git push --tags`.
|
|
162
|
+
6. Verify on <https://rubygems.org/gems/logstash-input-s3-sns-sqs>.
|
|
163
|
+
|
|
164
|
+
**Never** run `publish_gem` from macOS host Ruby — the resulting gem may miss
|
|
165
|
+
JRuby-specific vendor artefacts.
|
|
166
|
+
|
|
167
|
+
---
|
|
168
|
+
|
|
169
|
+
## 5. Code conventions & gotchas
|
|
170
|
+
|
|
171
|
+
- **JRuby-only Ruby features.** The code uses `java_import` and Java `Zlib` for
|
|
172
|
+
gzip perf. Do not switch to pure-Ruby gzip.
|
|
173
|
+
- **Ruby version drift.** JRuby 9.4+ tracks Ruby 3.1+. `File.exists?` (with `s`)
|
|
174
|
+
was removed. Always use `File.exist?`. This bit us in issue #94 / PR #95.
|
|
175
|
+
- **Threading.** Consumer threads are managed by hand plus `concurrent-ruby`.
|
|
176
|
+
Any exception must be caught and logged; otherwise the watcher will restart
|
|
177
|
+
the thread forever. Do not swallow errors silently.
|
|
178
|
+
- **AWS SDK v3.** The plugin migrated from SDK v2 to v3 in 2.2.0. Any new AWS
|
|
179
|
+
call must go through `S3::ClientFactory` (per-bucket region resolution).
|
|
180
|
+
- **Backwards config compat.** Deprecated flat `s3_*` options must still merge
|
|
181
|
+
into `s3_options_by_bucket` (see CHANGELOG 2.0.0). Don’t remove without a
|
|
182
|
+
major version bump.
|
|
183
|
+
- **UTF-8.** Log lines can contain broken UTF-8. `LogProcessor` uses
|
|
184
|
+
`scrub`-style handling — preserve it (see 1.0.2, 2.1.2).
|
|
185
|
+
- **No file deletion from S3 by default.** The plugin intentionally leaves
|
|
186
|
+
objects in place (users rely on S3 Lifecycle). `move_to_bucket` (PR #82)
|
|
187
|
+
is the opt-in.
|
|
188
|
+
|
|
189
|
+
---
|
|
190
|
+
|
|
191
|
+
## 6. Reviewing PRs
|
|
192
|
+
|
|
193
|
+
Recommended checklist before merging any PR:
|
|
194
|
+
|
|
195
|
+
1. `git fetch origin pull/<N>/head:pr-<N> && git checkout pr-<N>`
|
|
196
|
+
2. Read the diff top-to-bottom. Prefer minimal, focused PRs.
|
|
197
|
+
3. Confirm compatibility with **JRuby 9.4+ / Ruby 3.1+** (no `File.exists?`,
|
|
198
|
+
no `Fixnum`, no keyword-args regressions).
|
|
199
|
+
4. Run `bundle exec rspec` in a JRuby container (§3.1).
|
|
200
|
+
5. `--config.test_and_exit` smoke test inside `docker.elastic.co/logstash/logstash:9.4.4` (§3.4).
|
|
201
|
+
6. For AWS-touching changes: eyeball `S3::ClientFactory` and `SQS::Poller` for
|
|
202
|
+
missed error rescues.
|
|
203
|
+
7. Update `CHANGELOG.md` on merge if the PR did not.
|
|
204
|
+
|
|
205
|
+
---
|
|
206
|
+
|
|
207
|
+
## 7. Hard rules for agents
|
|
208
|
+
|
|
209
|
+
- **Never** commit unless the user explicitly asks.
|
|
210
|
+
- **Never** bump the gem version or edit `CHANGELOG.md` unless doing a release.
|
|
211
|
+
- **Never** re-introduce `logstash-mixin-aws` — the mixin is intentionally
|
|
212
|
+
inlined.
|
|
213
|
+
- **Never** run `publish_gem` — that is a maintainer action.
|
|
214
|
+
- **Never** use MRI Ruby to run tests; results are meaningless.
|
|
215
|
+
- Prefer podman over docker on this machine (`docker` is not installed;
|
|
216
|
+
`podman` is at `/opt/homebrew/bin/podman`, machine `podman-machine-default`
|
|
217
|
+
is running).
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
|
-
##2.2.
|
|
2
|
-
|
|
1
|
+
##2.2.4
|
|
2
|
+
- BUGFIX: Exclude `vendor/bundle/**` from the packaged gem. 2.2.3 accidentally shipped 63 MB of bundler-installed dev dependencies because the `s.files` glob included the whole `vendor/` tree. 2.2.3 is yanked from RubyGems. 2.2.4 has the same code fix as 2.2.3 (see below), just correctly packaged.
|
|
3
|
+
|
|
4
|
+
##2.2.3 (YANKED - packaging error, use 2.2.4)
|
|
5
|
+
- BUGFIX: `File.exists?` -> `File.exist?` for Ruby 3.2+ / JRuby 10 (Logstash 9.4.x). Fixes #93, #94. Thanks @ashleygould (PR #95).
|
|
6
|
+
- CHORE: Add `required_ruby_version = '>= 3.1'` documenting JRuby 9.4+ baseline.
|
|
7
|
+
- SECURITY note: CVE-2026-54904/54905/54906 in concurrent-ruby are not exploitable via this plugin (we only use `Concurrent::AtomicBoolean`). No dependency floor is bumped because logstash-core pins concurrent-ruby itself.
|
|
8
|
+
|
|
3
9
|
##2.2.2
|
|
4
10
|
Fix gzip validation not to return true if only file extension is gz
|
|
5
11
|
Make logstash 8/9 compatible
|
|
@@ -35,7 +35,7 @@ class S3Downloader
|
|
|
35
35
|
end
|
|
36
36
|
|
|
37
37
|
def cleanup_local_object(record)
|
|
38
|
-
FileUtils.remove_entry_secure(record[:local_file], true) if ::File.
|
|
38
|
+
FileUtils.remove_entry_secure(record[:local_file], true) if ::File.exist?(record[:local_file])
|
|
39
39
|
rescue Exception => e
|
|
40
40
|
@logger.warn("Could not delete file", :file => record[:local_file], :error => e)
|
|
41
41
|
end
|
|
@@ -329,72 +329,25 @@ class LogStash::Inputs::S3SNSSQS < LogStash::Inputs::Threadable
|
|
|
329
329
|
# --- END plugin interface ------------------------------------------#
|
|
330
330
|
|
|
331
331
|
private
|
|
332
|
-
Aws.config.update({
|
|
333
|
-
http_open_timeout: 60,
|
|
334
|
-
http_read_timeout: 120,
|
|
335
|
-
retry_limit: 5, # Increase the retry limit
|
|
336
|
-
retry_backoff: lambda { |c| sleep(2 ** c.retries) } # Exponential backoff
|
|
337
|
-
})
|
|
338
|
-
|
|
339
332
|
def run_worker_thread(queue, thread_id)
|
|
340
333
|
Thread.new do
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
completed = catch(:skip_delete) do
|
|
352
|
-
@logger.debug("[#{Thread.current[:name]}] processing record: #{record}")
|
|
353
|
-
process(record, queue)
|
|
354
|
-
end
|
|
355
|
-
@s3_downloader.cleanup_local_object(record)
|
|
356
|
-
# re-throw if necessary:
|
|
357
|
-
throw :skip_delete unless completed
|
|
358
|
-
@s3_downloader.move_s3object(record)
|
|
359
|
-
@s3_downloader.cleanup_s3object(record)
|
|
360
|
-
end
|
|
361
|
-
rescue Seahorse::Client::NetworkingError => e
|
|
362
|
-
@logger.error("[#{Thread.current[:name]}] Networking error inside poller loop: #{e.message}")
|
|
363
|
-
@retry_count ||= 0
|
|
364
|
-
if @retry_count < 5
|
|
365
|
-
@retry_count += 1
|
|
366
|
-
sleep(2 ** @retry_count) # Exponential backoff
|
|
367
|
-
logger.info("[#{Thread.current[:name]}] Retry #{retry_count} of 5 for networking error")
|
|
368
|
-
retry
|
|
369
|
-
else
|
|
370
|
-
@logger.error("[#{Thread.current[:name]}] Exceeded retry limit for networking error: #{e.message}")
|
|
371
|
-
@received_stop.make_true
|
|
372
|
-
raise
|
|
373
|
-
end
|
|
374
|
-
rescue StandardError => e
|
|
375
|
-
@logger.error("[#{Thread.current[:name]}] Error processing record inside poller loop: #{e.message}")
|
|
376
|
-
@logger.error(e.backtrace.join("\n"))
|
|
377
|
-
@received_stop.make_true
|
|
378
|
-
raise
|
|
334
|
+
LogStash::Util.set_thread_name("Worker #{@id}/#{thread_id}")
|
|
335
|
+
@logger.info("[#{Thread.current[:name]}] started (#{Time.now})") #PROFILING
|
|
336
|
+
temporary_directory = Dir.mktmpdir("#{@temporary_directory}/")
|
|
337
|
+
@sqs_poller.run do |record|
|
|
338
|
+
throw :skip_delete if stop?
|
|
339
|
+
# record is a valid object with the keys ":bucket", ":key", ":size"
|
|
340
|
+
record[:local_file] = File.join(temporary_directory, File.basename(record[:key]))
|
|
341
|
+
if @s3_downloader.copy_s3object_to_disk(record)
|
|
342
|
+
completed = catch(:skip_delete) do
|
|
343
|
+
process(record, queue)
|
|
379
344
|
end
|
|
345
|
+
@s3_downloader.cleanup_local_object(record)
|
|
346
|
+
# re-throw if necessary:
|
|
347
|
+
throw :skip_delete unless completed
|
|
348
|
+
@s3_downloader.move_s3object(record)
|
|
349
|
+
@s3_downloader.cleanup_s3object(record)
|
|
380
350
|
end
|
|
381
|
-
rescue Seahorse::Client::NetworkingError => e
|
|
382
|
-
@logger.error("Networking error in thread management: #{e.message}")
|
|
383
|
-
@retry_count ||= 0
|
|
384
|
-
if @retry_count < 5
|
|
385
|
-
@retry_count += 1
|
|
386
|
-
sleep(2 ** @retry_count) # Exponential backoff
|
|
387
|
-
logger.info("[#{Thread.current[:name]}] Retry #{retry_count} of 5 for networking error")
|
|
388
|
-
retry
|
|
389
|
-
else
|
|
390
|
-
@logger.error("Exceeded retry limit for networking error: #{e.message}")
|
|
391
|
-
@received_stop.make_true
|
|
392
|
-
raise
|
|
393
|
-
end
|
|
394
|
-
rescue StandardError => e
|
|
395
|
-
@logger.error("Unexpected error in thread management: #{e.message}")
|
|
396
|
-
@received_stop.make_true
|
|
397
|
-
raise
|
|
398
351
|
end
|
|
399
352
|
end
|
|
400
353
|
end
|
|
@@ -38,7 +38,7 @@ module LogStash::PluginMixins::AwsConfig::V2
|
|
|
38
38
|
if @use_aws_bundled_ca
|
|
39
39
|
aws_core_library = Gem.loaded_specs['aws-sdk-core']&.full_gem_path or fail("AWS Core library not available")
|
|
40
40
|
opts[:ssl_ca_bundle] = File.expand_path('ca-bundle.crt', aws_core_library).tap do |aws_core_ca_bundle|
|
|
41
|
-
fail("AWS Core CA bundle not found") unless File.
|
|
41
|
+
fail("AWS Core CA bundle not found") unless File.exist?(aws_core_ca_bundle)
|
|
42
42
|
end
|
|
43
43
|
end
|
|
44
44
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Gem::Specification.new do |s|
|
|
2
2
|
s.name = 'logstash-input-s3-sns-sqs'
|
|
3
|
-
s.version = '2.2.
|
|
3
|
+
s.version = '2.2.4'
|
|
4
4
|
s.licenses = ['Apache-2.0']
|
|
5
5
|
s.summary = "Get logs from AWS s3 buckets as issued by an object-created event via sns -> sqs."
|
|
6
6
|
s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
|
|
@@ -9,8 +9,15 @@ Gem::Specification.new do |s|
|
|
|
9
9
|
s.homepage = "https://github.com/cherweg/logstash-input-s3-sns-sqs"
|
|
10
10
|
s.require_paths = ["lib"]
|
|
11
11
|
|
|
12
|
+
# JRuby 9.4+ tracks Ruby 3.1+. Logstash 8/9 ships JRuby 9.4 or 10.
|
|
13
|
+
s.required_ruby_version = '>= 3.1'
|
|
14
|
+
|
|
12
15
|
# Files
|
|
13
|
-
|
|
16
|
+
# NOTE: We deliberately exclude vendor/bundle/** (bundler-installed dev gems
|
|
17
|
+
# end up there when developers run `bundle install --path vendor/bundle`).
|
|
18
|
+
# Including them ballooned 2.2.3 to 63 MB — see CHANGELOG. vendor/ is still
|
|
19
|
+
# included for future jar-dependencies (e.g. vendor/jar-dependencies/**).
|
|
20
|
+
s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT'].reject { |f| f.start_with?('vendor/bundle/') }
|
|
14
21
|
|
|
15
22
|
# Tests
|
|
16
23
|
s.test_files = s.files.grep(%r{^(test|spec|features)/})
|
|
@@ -20,6 +27,10 @@ Gem::Specification.new do |s|
|
|
|
20
27
|
|
|
21
28
|
# Gem dependencies
|
|
22
29
|
s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.0"
|
|
30
|
+
# Note: cannot pin a higher floor because logstash-core itself constrains
|
|
31
|
+
# concurrent-ruby (currently < 1.1.10). CVE-2026-54904/54905/54906 in
|
|
32
|
+
# concurrent-ruby are not exploitable via this plugin: we only use
|
|
33
|
+
# Concurrent::AtomicBoolean, not AtomicReference or ReadWriteLock.
|
|
23
34
|
s.add_runtime_dependency "concurrent-ruby"
|
|
24
35
|
s.add_runtime_dependency 'stud', '>= 0.0.22'
|
|
25
36
|
s.add_runtime_dependency "logstash-codec-json"
|
metadata
CHANGED
|
@@ -1,234 +1,233 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: logstash-input-s3-sns-sqs
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 2.2.
|
|
4
|
+
version: 2.2.4
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Christian Herweg
|
|
8
|
-
autorequire:
|
|
9
8
|
bindir: bin
|
|
10
9
|
cert_chain: []
|
|
11
|
-
date:
|
|
10
|
+
date: 2026-07-31 00:00:00.000000000 Z
|
|
12
11
|
dependencies:
|
|
13
12
|
- !ruby/object:Gem::Dependency
|
|
13
|
+
name: logstash-core-plugin-api
|
|
14
14
|
requirement: !ruby/object:Gem::Requirement
|
|
15
15
|
requirements:
|
|
16
16
|
- - "~>"
|
|
17
17
|
- !ruby/object:Gem::Version
|
|
18
18
|
version: '2.0'
|
|
19
|
-
name: logstash-core-plugin-api
|
|
20
|
-
prerelease: false
|
|
21
19
|
type: :runtime
|
|
20
|
+
prerelease: false
|
|
22
21
|
version_requirements: !ruby/object:Gem::Requirement
|
|
23
22
|
requirements:
|
|
24
23
|
- - "~>"
|
|
25
24
|
- !ruby/object:Gem::Version
|
|
26
25
|
version: '2.0'
|
|
27
26
|
- !ruby/object:Gem::Dependency
|
|
27
|
+
name: concurrent-ruby
|
|
28
28
|
requirement: !ruby/object:Gem::Requirement
|
|
29
29
|
requirements:
|
|
30
30
|
- - ">="
|
|
31
31
|
- !ruby/object:Gem::Version
|
|
32
32
|
version: '0'
|
|
33
|
-
name: concurrent-ruby
|
|
34
|
-
prerelease: false
|
|
35
33
|
type: :runtime
|
|
34
|
+
prerelease: false
|
|
36
35
|
version_requirements: !ruby/object:Gem::Requirement
|
|
37
36
|
requirements:
|
|
38
37
|
- - ">="
|
|
39
38
|
- !ruby/object:Gem::Version
|
|
40
39
|
version: '0'
|
|
41
40
|
- !ruby/object:Gem::Dependency
|
|
41
|
+
name: stud
|
|
42
42
|
requirement: !ruby/object:Gem::Requirement
|
|
43
43
|
requirements:
|
|
44
44
|
- - ">="
|
|
45
45
|
- !ruby/object:Gem::Version
|
|
46
46
|
version: 0.0.22
|
|
47
|
-
name: stud
|
|
48
|
-
prerelease: false
|
|
49
47
|
type: :runtime
|
|
48
|
+
prerelease: false
|
|
50
49
|
version_requirements: !ruby/object:Gem::Requirement
|
|
51
50
|
requirements:
|
|
52
51
|
- - ">="
|
|
53
52
|
- !ruby/object:Gem::Version
|
|
54
53
|
version: 0.0.22
|
|
55
54
|
- !ruby/object:Gem::Dependency
|
|
55
|
+
name: logstash-codec-json
|
|
56
56
|
requirement: !ruby/object:Gem::Requirement
|
|
57
57
|
requirements:
|
|
58
58
|
- - ">="
|
|
59
59
|
- !ruby/object:Gem::Version
|
|
60
60
|
version: '0'
|
|
61
|
-
name: logstash-codec-json
|
|
62
|
-
prerelease: false
|
|
63
61
|
type: :runtime
|
|
62
|
+
prerelease: false
|
|
64
63
|
version_requirements: !ruby/object:Gem::Requirement
|
|
65
64
|
requirements:
|
|
66
65
|
- - ">="
|
|
67
66
|
- !ruby/object:Gem::Version
|
|
68
67
|
version: '0'
|
|
69
68
|
- !ruby/object:Gem::Dependency
|
|
69
|
+
name: logstash-codec-plain
|
|
70
70
|
requirement: !ruby/object:Gem::Requirement
|
|
71
71
|
requirements:
|
|
72
72
|
- - ">="
|
|
73
73
|
- !ruby/object:Gem::Version
|
|
74
74
|
version: '0'
|
|
75
|
-
name: logstash-codec-plain
|
|
76
|
-
prerelease: false
|
|
77
75
|
type: :runtime
|
|
76
|
+
prerelease: false
|
|
78
77
|
version_requirements: !ruby/object:Gem::Requirement
|
|
79
78
|
requirements:
|
|
80
79
|
- - ">="
|
|
81
80
|
- !ruby/object:Gem::Version
|
|
82
81
|
version: '0'
|
|
83
82
|
- !ruby/object:Gem::Dependency
|
|
83
|
+
name: aws-sdk-core
|
|
84
84
|
requirement: !ruby/object:Gem::Requirement
|
|
85
85
|
requirements:
|
|
86
86
|
- - "~>"
|
|
87
87
|
- !ruby/object:Gem::Version
|
|
88
88
|
version: '3'
|
|
89
|
-
name: aws-sdk-core
|
|
90
|
-
prerelease: false
|
|
91
89
|
type: :runtime
|
|
90
|
+
prerelease: false
|
|
92
91
|
version_requirements: !ruby/object:Gem::Requirement
|
|
93
92
|
requirements:
|
|
94
93
|
- - "~>"
|
|
95
94
|
- !ruby/object:Gem::Version
|
|
96
95
|
version: '3'
|
|
97
96
|
- !ruby/object:Gem::Dependency
|
|
97
|
+
name: aws-sdk-s3
|
|
98
98
|
requirement: !ruby/object:Gem::Requirement
|
|
99
99
|
requirements:
|
|
100
100
|
- - ">="
|
|
101
101
|
- !ruby/object:Gem::Version
|
|
102
102
|
version: '0'
|
|
103
|
-
name: aws-sdk-s3
|
|
104
|
-
prerelease: false
|
|
105
103
|
type: :runtime
|
|
104
|
+
prerelease: false
|
|
106
105
|
version_requirements: !ruby/object:Gem::Requirement
|
|
107
106
|
requirements:
|
|
108
107
|
- - ">="
|
|
109
108
|
- !ruby/object:Gem::Version
|
|
110
109
|
version: '0'
|
|
111
110
|
- !ruby/object:Gem::Dependency
|
|
111
|
+
name: aws-sdk-sqs
|
|
112
112
|
requirement: !ruby/object:Gem::Requirement
|
|
113
113
|
requirements:
|
|
114
114
|
- - ">="
|
|
115
115
|
- !ruby/object:Gem::Version
|
|
116
116
|
version: '0'
|
|
117
|
-
name: aws-sdk-sqs
|
|
118
|
-
prerelease: false
|
|
119
117
|
type: :runtime
|
|
118
|
+
prerelease: false
|
|
120
119
|
version_requirements: !ruby/object:Gem::Requirement
|
|
121
120
|
requirements:
|
|
122
121
|
- - ">="
|
|
123
122
|
- !ruby/object:Gem::Version
|
|
124
123
|
version: '0'
|
|
125
124
|
- !ruby/object:Gem::Dependency
|
|
125
|
+
name: aws-sdk-sns
|
|
126
126
|
requirement: !ruby/object:Gem::Requirement
|
|
127
127
|
requirements:
|
|
128
128
|
- - ">="
|
|
129
129
|
- !ruby/object:Gem::Version
|
|
130
130
|
version: '0'
|
|
131
|
-
name: aws-sdk-sns
|
|
132
|
-
prerelease: false
|
|
133
131
|
type: :runtime
|
|
132
|
+
prerelease: false
|
|
134
133
|
version_requirements: !ruby/object:Gem::Requirement
|
|
135
134
|
requirements:
|
|
136
135
|
- - ">="
|
|
137
136
|
- !ruby/object:Gem::Version
|
|
138
137
|
version: '0'
|
|
139
138
|
- !ruby/object:Gem::Dependency
|
|
139
|
+
name: aws-sdk-resourcegroups
|
|
140
140
|
requirement: !ruby/object:Gem::Requirement
|
|
141
141
|
requirements:
|
|
142
142
|
- - ">="
|
|
143
143
|
- !ruby/object:Gem::Version
|
|
144
144
|
version: '0'
|
|
145
|
-
name: aws-sdk-resourcegroups
|
|
146
|
-
prerelease: false
|
|
147
145
|
type: :runtime
|
|
146
|
+
prerelease: false
|
|
148
147
|
version_requirements: !ruby/object:Gem::Requirement
|
|
149
148
|
requirements:
|
|
150
149
|
- - ">="
|
|
151
150
|
- !ruby/object:Gem::Version
|
|
152
151
|
version: '0'
|
|
153
152
|
- !ruby/object:Gem::Dependency
|
|
153
|
+
name: logstash-codec-json_lines
|
|
154
154
|
requirement: !ruby/object:Gem::Requirement
|
|
155
155
|
requirements:
|
|
156
156
|
- - ">="
|
|
157
157
|
- !ruby/object:Gem::Version
|
|
158
158
|
version: '0'
|
|
159
|
-
name: logstash-codec-json_lines
|
|
160
|
-
prerelease: false
|
|
161
159
|
type: :development
|
|
160
|
+
prerelease: false
|
|
162
161
|
version_requirements: !ruby/object:Gem::Requirement
|
|
163
162
|
requirements:
|
|
164
163
|
- - ">="
|
|
165
164
|
- !ruby/object:Gem::Version
|
|
166
165
|
version: '0'
|
|
167
166
|
- !ruby/object:Gem::Dependency
|
|
167
|
+
name: logstash-codec-multiline
|
|
168
168
|
requirement: !ruby/object:Gem::Requirement
|
|
169
169
|
requirements:
|
|
170
170
|
- - ">="
|
|
171
171
|
- !ruby/object:Gem::Version
|
|
172
172
|
version: '0'
|
|
173
|
-
name: logstash-codec-multiline
|
|
174
|
-
prerelease: false
|
|
175
173
|
type: :development
|
|
174
|
+
prerelease: false
|
|
176
175
|
version_requirements: !ruby/object:Gem::Requirement
|
|
177
176
|
requirements:
|
|
178
177
|
- - ">="
|
|
179
178
|
- !ruby/object:Gem::Version
|
|
180
179
|
version: '0'
|
|
181
180
|
- !ruby/object:Gem::Dependency
|
|
181
|
+
name: logstash-codec-line
|
|
182
182
|
requirement: !ruby/object:Gem::Requirement
|
|
183
183
|
requirements:
|
|
184
184
|
- - ">="
|
|
185
185
|
- !ruby/object:Gem::Version
|
|
186
186
|
version: '0'
|
|
187
|
-
name: logstash-codec-line
|
|
188
|
-
prerelease: false
|
|
189
187
|
type: :development
|
|
188
|
+
prerelease: false
|
|
190
189
|
version_requirements: !ruby/object:Gem::Requirement
|
|
191
190
|
requirements:
|
|
192
191
|
- - ">="
|
|
193
192
|
- !ruby/object:Gem::Version
|
|
194
193
|
version: '0'
|
|
195
194
|
- !ruby/object:Gem::Dependency
|
|
195
|
+
name: logstash-devutils
|
|
196
196
|
requirement: !ruby/object:Gem::Requirement
|
|
197
197
|
requirements:
|
|
198
198
|
- - ">="
|
|
199
199
|
- !ruby/object:Gem::Version
|
|
200
200
|
version: 0.0.16
|
|
201
|
-
name: logstash-devutils
|
|
202
|
-
prerelease: false
|
|
203
201
|
type: :development
|
|
202
|
+
prerelease: false
|
|
204
203
|
version_requirements: !ruby/object:Gem::Requirement
|
|
205
204
|
requirements:
|
|
206
205
|
- - ">="
|
|
207
206
|
- !ruby/object:Gem::Version
|
|
208
207
|
version: 0.0.16
|
|
209
208
|
- !ruby/object:Gem::Dependency
|
|
209
|
+
name: logstash-input-generator
|
|
210
210
|
requirement: !ruby/object:Gem::Requirement
|
|
211
211
|
requirements:
|
|
212
212
|
- - ">="
|
|
213
213
|
- !ruby/object:Gem::Version
|
|
214
214
|
version: '0'
|
|
215
|
-
name: logstash-input-generator
|
|
216
|
-
prerelease: false
|
|
217
215
|
type: :development
|
|
216
|
+
prerelease: false
|
|
218
217
|
version_requirements: !ruby/object:Gem::Requirement
|
|
219
218
|
requirements:
|
|
220
219
|
- - ">="
|
|
221
220
|
- !ruby/object:Gem::Version
|
|
222
221
|
version: '0'
|
|
223
222
|
- !ruby/object:Gem::Dependency
|
|
223
|
+
name: timecop
|
|
224
224
|
requirement: !ruby/object:Gem::Requirement
|
|
225
225
|
requirements:
|
|
226
226
|
- - ">="
|
|
227
227
|
- !ruby/object:Gem::Version
|
|
228
228
|
version: '0'
|
|
229
|
-
name: timecop
|
|
230
|
-
prerelease: false
|
|
231
229
|
type: :development
|
|
230
|
+
prerelease: false
|
|
232
231
|
version_requirements: !ruby/object:Gem::Requirement
|
|
233
232
|
requirements:
|
|
234
233
|
- - ">="
|
|
@@ -242,6 +241,7 @@ executables: []
|
|
|
242
241
|
extensions: []
|
|
243
242
|
extra_rdoc_files: []
|
|
244
243
|
files:
|
|
244
|
+
- AGENTS.md
|
|
245
245
|
- CHANGELOG.md
|
|
246
246
|
- CONTRIBUTORS
|
|
247
247
|
- Gemfile
|
|
@@ -266,7 +266,6 @@ licenses:
|
|
|
266
266
|
metadata:
|
|
267
267
|
logstash_plugin: 'true'
|
|
268
268
|
logstash_group: input
|
|
269
|
-
post_install_message:
|
|
270
269
|
rdoc_options: []
|
|
271
270
|
require_paths:
|
|
272
271
|
- lib
|
|
@@ -274,15 +273,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
|
274
273
|
requirements:
|
|
275
274
|
- - ">="
|
|
276
275
|
- !ruby/object:Gem::Version
|
|
277
|
-
version: '
|
|
276
|
+
version: '3.1'
|
|
278
277
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
279
278
|
requirements:
|
|
280
|
-
- - "
|
|
279
|
+
- - ">="
|
|
281
280
|
- !ruby/object:Gem::Version
|
|
282
|
-
version:
|
|
281
|
+
version: '0'
|
|
283
282
|
requirements: []
|
|
284
|
-
rubygems_version: 3.
|
|
285
|
-
signing_key:
|
|
283
|
+
rubygems_version: 3.6.3
|
|
286
284
|
specification_version: 4
|
|
287
285
|
summary: Get logs from AWS s3 buckets as issued by an object-created event via sns
|
|
288
286
|
-> sqs.
|