logstash-logger 0.26.1 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (71) hide show
  1. checksums.yaml +4 -4
  2. data/.github/workflows/tests.yml +53 -0
  3. data/.gitignore +1 -0
  4. data/.rubocop.yml +85 -105
  5. data/Appraisals +6 -20
  6. data/CHANGELOG.md +61 -0
  7. data/Gemfile +2 -2
  8. data/README.md +12 -833
  9. data/docs/buffering.md +86 -0
  10. data/docs/customization.md +86 -0
  11. data/docs/outputs.md +65 -0
  12. data/docs/rails.md +349 -0
  13. data/docs/ssl.md +102 -0
  14. data/docs/troubleshooting.md +106 -0
  15. data/docs/usage.md +148 -0
  16. data/gemfiles/rails_8.0.gemfile +10 -0
  17. data/gemfiles/rails_8.1.gemfile +9 -0
  18. data/lib/logstash-logger/buffer.rb +15 -5
  19. data/lib/logstash-logger/configuration.rb +1 -2
  20. data/lib/logstash-logger/device/aws_stream.rb +96 -17
  21. data/lib/logstash-logger/device/base.rb +2 -2
  22. data/lib/logstash-logger/device/connectable.rb +30 -12
  23. data/lib/logstash-logger/device/file.rb +21 -4
  24. data/lib/logstash-logger/device/firehose.rb +6 -0
  25. data/lib/logstash-logger/device/http.rb +104 -0
  26. data/lib/logstash-logger/device/kafka.rb +182 -38
  27. data/lib/logstash-logger/device/kinesis.rb +10 -1
  28. data/lib/logstash-logger/device/redis.rb +8 -1
  29. data/lib/logstash-logger/device/tcp.rb +54 -12
  30. data/lib/logstash-logger/device.rb +24 -2
  31. data/lib/logstash-logger/formatter/base.rb +97 -11
  32. data/lib/logstash-logger/formatter/cee_syslog.rb +8 -11
  33. data/lib/logstash-logger/formatter/json.rb +13 -0
  34. data/lib/logstash-logger/formatter/json_lines.rb +13 -0
  35. data/lib/logstash-logger/formatter.rb +14 -6
  36. data/lib/logstash-logger/logger.rb +6 -19
  37. data/lib/logstash-logger/multi_logger.rb +2 -1
  38. data/lib/logstash-logger/railtie.rb +1 -1
  39. data/lib/logstash-logger/tagged_logging.rb +3 -1
  40. data/lib/logstash-logger/version.rb +1 -1
  41. data/logstash-logger.gemspec +8 -4
  42. data/spec/device/aws_stream_limits_spec.rb +219 -0
  43. data/spec/device/connectable_spec.rb +51 -0
  44. data/spec/device/file_spec.rb +65 -0
  45. data/spec/device/firehose_spec.rb +4 -1
  46. data/spec/device/http_spec.rb +176 -0
  47. data/spec/device/kafka_spec.rb +429 -14
  48. data/spec/device/kinesis_spec.rb +4 -1
  49. data/spec/device/tcp_spec.rb +94 -5
  50. data/spec/device/tcp_tls_integration_spec.rb +91 -0
  51. data/spec/device_spec.rb +13 -0
  52. data/spec/formatter/base_spec.rb +82 -1
  53. data/spec/formatter/cee_syslog_spec.rb +71 -19
  54. data/spec/formatter/framing_spec.rb +58 -0
  55. data/spec/formatter/json_lines_spec.rb +41 -0
  56. data/spec/formatter/json_spec.rb +49 -0
  57. data/spec/formatter_spec.rb +19 -2
  58. data/spec/logger_spec.rb +5 -5
  59. data/spec/multi_logger_spec.rb +16 -0
  60. data/spec/spec_helper.rb +7 -5
  61. data/spec/support/tls_helper.rb +30 -0
  62. data/spec/support/unserializable_value.rb +10 -0
  63. data/spec/tagged_logging_spec.rb +15 -0
  64. metadata +101 -16
  65. data/.travis.yml +0 -26
  66. data/gemfiles/rails_3.2.gemfile +0 -9
  67. data/gemfiles/rails_4.0.gemfile +0 -9
  68. data/gemfiles/rails_4.1.gemfile +0 -9
  69. data/gemfiles/rails_4.2.gemfile +0 -9
  70. data/gemfiles/rails_5.0.gemfile +0 -9
  71. data/gemfiles/rails_5.1.gemfile +0 -9
data/CHANGELOG.md CHANGED
@@ -1,3 +1,64 @@
1
+ ## 1.1.0
2
+
3
+ ### Security
4
+ - The TCP output with `ssl_enable: true` and no `ssl_context` accepted any server certificate, so anyone able to intercept the connection could read or alter the log stream. It now verifies the certificate and hostname. [GHSA-p7gr-ppfj-g45v](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-p7gr-ppfj-g45v)
5
+ - Logged content could break record framing: inject a line break into the `cee_syslog` header, or trigger an unframed `Failed to format log event` string that merged with the next record. [GHSA-88vf-w58m-gv7h](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-88vf-w58m-gv7h)
6
+ - One record that Kafka, Kinesis, Firehose or an HTTP collector always rejects could make the output hold back, duplicate or discard the other records buffered with it. [GHSA-qcm4-6phg-mhpf](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-qcm4-6phg-mhpf)
7
+
8
+ ### Breaking Changes
9
+ - Drop support for Ruby 3.2 and Rails 7.2, which are end of life. [#185](https://github.com/dwbutler/logstash-logger/pull/185)
10
+ - `ssl_enable: true` without an `ssl_context` now verifies the server certificate against the system trust store. For a self-signed or private-CA server, pass its certificate as `ssl_certificate`. Turning verification off requires an explicit `VERIFY_NONE` context. See [SSL Configuration](docs/ssl.md). [GHSA-p7gr-ppfj-g45v](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-p7gr-ppfj-g45v)
11
+ - `ssl_certificate` is now the certificate trusted for the server, as documented, and is checked when the logger is created. [GHSA-p7gr-ppfj-g45v](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-p7gr-ppfj-g45v)
12
+ - The `cee_syslog` header always uses this machine's hostname. A `host` set in logged content or in `customize_event` appears only in the JSON body. [GHSA-88vf-w58m-gv7h](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-88vf-w58m-gv7h)
13
+
14
+ ### Features
15
+ - `format: :ndjson` option for the HTTP device, to send newline-delimited JSON instead of a JSON array, and a `content_type` option. [#182](https://github.com/dwbutler/logstash-logger/pull/182) [#180](https://github.com/dwbutler/logstash-logger/issues/180)
16
+ - `max_record_bytes` option to drop any formatted record over a size limit, for every buffered output. Off by default. [GHSA-qcm4-6phg-mhpf](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-qcm4-6phg-mhpf)
17
+ - `aws_max_record_bytes` option for Kinesis streams configured for records larger than 1 MiB. [GHSA-qcm4-6phg-mhpf](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-qcm4-6phg-mhpf)
18
+ - The TCP output sends SNI for TLS connections to DNS hostnames. [GHSA-p7gr-ppfj-g45v](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-p7gr-ppfj-g45v)
19
+
20
+ ### Bug Fixes
21
+ - HTTP device raises on non-2xx responses, so failed uploads are retried instead of silently dropped. [#184](https://github.com/dwbutler/logstash-logger/pull/184) [#181](https://github.com/dwbutler/logstash-logger/issues/181)
22
+ - HTTP device drops only the records a collector rejects with 400, 413, 415 or 422. Other error statuses are retried. [GHSA-qcm4-6phg-mhpf](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-qcm4-6phg-mhpf)
23
+ - Kinesis and Firehose enforce the service's record and request limits, retry failed requests instead of treating them as delivered, and requeue throttled records without losing the batch. Kinesis `InternalFailure` records are retried. [GHSA-qcm4-6phg-mhpf](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-qcm4-6phg-mhpf)
24
+ - Kafka drops only records the broker always rejects, instead of retrying the whole batch until the buffer fills. [GHSA-qcm4-6phg-mhpf](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-qcm4-6phg-mhpf)
25
+ - The TCP output closes the underlying TCP socket when a TLS connection is closed or fails to set up. [GHSA-p7gr-ppfj-g45v](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-p7gr-ppfj-g45v)
26
+ - A record that cannot be formatted is replaced by a placeholder event tagged `_logstashlogger_format_failure`, framed like any other record. [GHSA-88vf-w58m-gv7h](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-88vf-w58m-gv7h)
27
+ - An `@timestamp` in logged content that is not a valid time no longer fails the record. [GHSA-88vf-w58m-gv7h](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-88vf-w58m-gv7h)
28
+ - The `cee_syslog` formatter no longer shares the program name between concurrent log calls. [GHSA-88vf-w58m-gv7h](https://github.com/dwbutler/logstash-logger/security/advisories/GHSA-88vf-w58m-gv7h)
29
+
30
+ ### Other
31
+ - CI covers Ruby 3.3, 3.4, 4.0 and JRuby 10 with Rails 8.0 and 8.1, and uploads coverage to Codecov. [#183](https://github.com/dwbutler/logstash-logger/pull/183) [#185](https://github.com/dwbutler/logstash-logger/pull/185)
32
+ - Fix an order-dependent Kafka spec. [#186](https://github.com/dwbutler/logstash-logger/pull/186)
33
+
34
+ ## 1.0.0
35
+
36
+ ### Breaking Changes
37
+ - Remove deprecated features (deprecated constructor, Poseidon Kafka implementation). [#177](https://github.com/dwbutler/logstash-logger/pull/177)
38
+ - Drops support for older Ruby/Rails versions. Configures modern Ruby and Rails. [#176](https://github.com/dwbutler/logstash-logger/pull/176)
39
+
40
+ ### Features
41
+ - New Kafka device implementation using `ruby-kafka` gem, replacing deprecated Poseidon. Includes topic validation, buffer group configuration, and improved error handling. [#175](https://github.com/dwbutler/logstash-logger/pull/175)
42
+ - Add buffered HTTP device to write to Logstash HTTP input. [#163](https://github.com/dwbutler/logstash-logger/pull/163) [#135](https://github.com/dwbutler/logstash-logger/issues/135)
43
+ - Support for `shift_age` and `shift_size` options in File device. [#164](https://github.com/dwbutler/logstash-logger/issues/164)
44
+ - Configure `buffer_max_items` for Connectable devices using URI query string options. [#176](https://github.com/dwbutler/logstash-logger/pull/176)
45
+ - Wrap hostname into a hash to ensure compatibility with Elastic >= 8. [#172](https://github.com/dwbutler/logstash-logger/pull/172)
46
+ - Put `@timestamp` first in JSON output. [#174](https://github.com/dwbutler/logstash-logger/pull/174) [#168](https://github.com/dwbutler/logstash-logger/issues/168)
47
+ - Instantiate custom formatter classes with `customize_event` keyword arg. [#155](https://github.com/dwbutler/logstash-logger/pull/155)
48
+ - Allow `.reset` on MultiLogger. [#153](https://github.com/dwbutler/logstash-logger/pull/153) [#151](https://github.com/dwbutler/logstash-logger/issues/151)
49
+
50
+ ### Bug Fixes
51
+ - Fix JRuby compatibility issues. [#179](https://github.com/dwbutler/logstash-logger/pull/179)
52
+ - Handle non-string tags in tagged logging. [#173](https://github.com/dwbutler/logstash-logger/pull/173)
53
+ - Allow passing integers as tags to `TaggedLogging::Formatter`. [#152](https://github.com/dwbutler/logstash-logger/pull/152)
54
+ - Fix alias in MultiLogger. [#161](https://github.com/dwbutler/logstash-logger/pull/161)
55
+ - Gracefully handle JSON `Encoding::UndefinedConversionError`. [#165](https://github.com/dwbutler/logstash-logger/pull/165) [#158](https://github.com/dwbutler/logstash-logger/issues/158)
56
+ - Fix argument error for unknown keyword `:sync` with Redis connection. [#171](https://github.com/dwbutler/logstash-logger/pull/171)
57
+
58
+ ### Other
59
+ - Move documentation from README to `docs/` directory. [#178](https://github.com/dwbutler/logstash-logger/pull/178)
60
+ - Migrate CI to GitHub Actions.
61
+
1
62
  ## 0.26.1
2
63
 
3
64
  - Fix: Support for SSL with the TCP device without an SSL context in Ruby
data/Gemfile CHANGED
@@ -2,5 +2,5 @@ source 'https://rubygems.org'
2
2
 
3
3
  gemspec
4
4
 
5
- gem 'codecov', require: false, group: :test
6
- gem "codeclimate-test-reporter", group: :test, require: nil
5
+ gem "simplecov", ">= 0.22", group: :test, require: false
6
+ gem "simplecov-cobertura", group: :test, require: false