logstash-logger-p 0.26.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (94) hide show
  1. checksums.yaml +7 -0
  2. data/.gitignore +21 -0
  3. data/.rspec +3 -0
  4. data/.rubocop.yml +1156 -0
  5. data/.travis.yml +26 -0
  6. data/Appraisals +23 -0
  7. data/CHANGELOG.md +199 -0
  8. data/Gemfile +6 -0
  9. data/LICENSE.txt +22 -0
  10. data/README.md +880 -0
  11. data/Rakefile +23 -0
  12. data/gemfiles/rails_3.2.gemfile +9 -0
  13. data/gemfiles/rails_4.0.gemfile +9 -0
  14. data/gemfiles/rails_4.1.gemfile +9 -0
  15. data/gemfiles/rails_4.2.gemfile +9 -0
  16. data/gemfiles/rails_5.0.gemfile +9 -0
  17. data/gemfiles/rails_5.1.gemfile +9 -0
  18. data/lib/logstash-logger/buffer.rb +336 -0
  19. data/lib/logstash-logger/configuration.rb +29 -0
  20. data/lib/logstash-logger/device/aws_stream.rb +94 -0
  21. data/lib/logstash-logger/device/balancer.rb +40 -0
  22. data/lib/logstash-logger/device/base.rb +73 -0
  23. data/lib/logstash-logger/device/connectable.rb +131 -0
  24. data/lib/logstash-logger/device/file.rb +23 -0
  25. data/lib/logstash-logger/device/firehose.rb +42 -0
  26. data/lib/logstash-logger/device/io.rb +11 -0
  27. data/lib/logstash-logger/device/kafka.rb +57 -0
  28. data/lib/logstash-logger/device/kinesis.rb +44 -0
  29. data/lib/logstash-logger/device/multi_delegator.rb +36 -0
  30. data/lib/logstash-logger/device/redis.rb +76 -0
  31. data/lib/logstash-logger/device/socket.rb +21 -0
  32. data/lib/logstash-logger/device/stderr.rb +13 -0
  33. data/lib/logstash-logger/device/stdout.rb +14 -0
  34. data/lib/logstash-logger/device/tcp.rb +86 -0
  35. data/lib/logstash-logger/device/udp.rb +12 -0
  36. data/lib/logstash-logger/device/unix.rb +18 -0
  37. data/lib/logstash-logger/device.rb +67 -0
  38. data/lib/logstash-logger/formatter/base.rb +73 -0
  39. data/lib/logstash-logger/formatter/cee.rb +11 -0
  40. data/lib/logstash-logger/formatter/cee_syslog.rb +22 -0
  41. data/lib/logstash-logger/formatter/json.rb +11 -0
  42. data/lib/logstash-logger/formatter/json_lines.rb +11 -0
  43. data/lib/logstash-logger/formatter/logstash_event.rb +6 -0
  44. data/lib/logstash-logger/formatter.rb +51 -0
  45. data/lib/logstash-logger/logger.rb +106 -0
  46. data/lib/logstash-logger/multi_logger.rb +153 -0
  47. data/lib/logstash-logger/railtie.rb +51 -0
  48. data/lib/logstash-logger/silenced_logging.rb +83 -0
  49. data/lib/logstash-logger/tagged_logging.rb +40 -0
  50. data/lib/logstash-logger/version.rb +3 -0
  51. data/lib/logstash-logger.rb +11 -0
  52. data/logstash-logger.gemspec +39 -0
  53. data/samples/example.crt +16 -0
  54. data/samples/example.key +15 -0
  55. data/samples/file.conf +11 -0
  56. data/samples/redis.conf +12 -0
  57. data/samples/ssl.conf +15 -0
  58. data/samples/syslog.conf +10 -0
  59. data/samples/tcp.conf +11 -0
  60. data/samples/udp.conf +11 -0
  61. data/samples/unix.conf +11 -0
  62. data/spec/configuration_spec.rb +27 -0
  63. data/spec/constructor_spec.rb +30 -0
  64. data/spec/device/balancer_spec.rb +31 -0
  65. data/spec/device/connectable_spec.rb +74 -0
  66. data/spec/device/file_spec.rb +15 -0
  67. data/spec/device/firehose_spec.rb +41 -0
  68. data/spec/device/io_spec.rb +13 -0
  69. data/spec/device/kafka_spec.rb +32 -0
  70. data/spec/device/kinesis_spec.rb +41 -0
  71. data/spec/device/multi_delegator_spec.rb +31 -0
  72. data/spec/device/redis_spec.rb +52 -0
  73. data/spec/device/socket_spec.rb +15 -0
  74. data/spec/device/stderr_spec.rb +16 -0
  75. data/spec/device/stdout_spec.rb +31 -0
  76. data/spec/device/tcp_spec.rb +120 -0
  77. data/spec/device/udp_spec.rb +9 -0
  78. data/spec/device/unix_spec.rb +23 -0
  79. data/spec/device_spec.rb +97 -0
  80. data/spec/formatter/base_spec.rb +125 -0
  81. data/spec/formatter/cee_spec.rb +15 -0
  82. data/spec/formatter/cee_syslog_spec.rb +43 -0
  83. data/spec/formatter/json_lines_spec.rb +14 -0
  84. data/spec/formatter/json_spec.rb +10 -0
  85. data/spec/formatter/logstash_event_spec.rb +10 -0
  86. data/spec/formatter_spec.rb +79 -0
  87. data/spec/logger_spec.rb +128 -0
  88. data/spec/multi_logger_spec.rb +59 -0
  89. data/spec/rails_spec.rb +91 -0
  90. data/spec/silenced_logging_spec.rb +31 -0
  91. data/spec/spec_helper.rb +111 -0
  92. data/spec/syslog_spec.rb +32 -0
  93. data/spec/tagged_logging_spec.rb +32 -0
  94. metadata +335 -0
data/README.md ADDED
@@ -0,0 +1,880 @@
1
+ # LogStashLogger
2
+ [![Build Status](https://travis-ci.org/dwbutler/logstash-logger.svg?branch=master)](https://travis-ci.org/dwbutler/logstash-logger) [![Code Climate](https://codeclimate.com/github/dwbutler/logstash-logger/badges/gpa.svg)](https://codeclimate.com/github/dwbutler/logstash-logger) [![codecov.io](http://codecov.io/github/dwbutler/logstash-logger/coverage.svg?branch=master)](http://codecov.io/github/dwbutler/logstash-logger?branch=master) [![Gem Version](https://badge.fury.io/rb/logstash-logger.svg)](https://badge.fury.io/rb/logstash-logger)
3
+
4
+ LogStashLogger extends Ruby's `Logger` class to log directly to
5
+ [Logstash](https://www.elastic.co/products/logstash).
6
+ It supports writing to various outputs in logstash JSON format. This is an improvement over
7
+ writing to a file or syslog since Logstash can receive the structured data directly.
8
+
9
+ ## Features
10
+
11
+ * Can write directly to a logstash listener over a UDP or TCP/SSL connection.
12
+ * Can write to a file, Redis, Kafka, Kinesis, Firehose, a unix socket, syslog, stdout, or stderr.
13
+ * Logger can take a string message, a hash, a `LogStash::Event`, an object, or a JSON string as input.
14
+ * Events are automatically populated with message, timestamp, host, and severity.
15
+ * Writes in logstash JSON format, but supports other formats as well.
16
+ * Can write to multiple outputs.
17
+ * Log messages are buffered and automatically re-sent if there is a connection problem.
18
+ * Easily integrates with Rails via configuration.
19
+
20
+ ## Installation
21
+
22
+ Add this line to your application's Gemfile:
23
+
24
+ gem 'logstash-logger'
25
+
26
+ And then execute:
27
+
28
+ $ bundle
29
+
30
+ Or install it yourself as:
31
+
32
+ $ gem install logstash-logger
33
+
34
+ ## Usage Examples
35
+
36
+ ```ruby
37
+ require 'logstash-logger'
38
+
39
+ # Defaults to UDP on 0.0.0.0
40
+ logger = LogStashLogger.new(port: 5228)
41
+
42
+ # Specify host and type (UDP or TCP) explicitly
43
+ udp_logger = LogStashLogger.new(type: :udp, host: 'localhost', port: 5228)
44
+ tcp_logger = LogStashLogger.new(type: :tcp, host: 'localhost', port: 5229)
45
+
46
+ # Other types of loggers
47
+ file_logger = LogStashLogger.new(type: :file, path: 'log/development.log', sync: true)
48
+ unix_logger = LogStashLogger.new(type: :unix, path: '/tmp/sock')
49
+ syslog_logger = LogStashLogger.new(type: :syslog)
50
+ redis_logger = LogStashLogger.new(type: :redis)
51
+ kafka_logger = LogStashLogger.new(type: :kafka)
52
+ stdout_logger = LogStashLogger.new(type: :stdout)
53
+ stderr_logger = LogStashLogger.new(type: :stderr)
54
+ io_logger = LogStashLogger.new(type: :io, io: io)
55
+
56
+ # Use a different formatter
57
+ cee_logger = LogStashLogger.new(
58
+ type: :tcp,
59
+ host: 'logsene-receiver-syslog.sematext.com',
60
+ port: 514,
61
+ formatter: :cee_syslog
62
+ )
63
+
64
+ custom_formatted_logger = LogStashLogger.new(
65
+ type: :redis,
66
+ formatter: MyCustomFormatter
67
+ )
68
+
69
+ lambda_formatted_logger = LogStashLogger.new(
70
+ type: :stdout,
71
+ formatter: ->(severity, time, progname, msg) { "[#{progname}] #{msg}" }
72
+ )
73
+
74
+ ruby_default_formatter_logger = LogStashLogger.new(
75
+ type: :file,
76
+ path: 'log/development.log',
77
+ formatter: ::Logger::Formatter
78
+ )
79
+
80
+ # Send messages to multiple outputs. Each output will have the same format.
81
+ # Syslog cannot be an output because it requires a separate logger.
82
+ multi_delegating_logger = LogStashLogger.new(
83
+ type: :multi_delegator,
84
+ outputs: [
85
+ { type: :file, path: 'log/development.log' },
86
+ { type: :udp, host: 'localhost', port: 5228 }
87
+ ])
88
+
89
+ # Balance messages between several outputs.
90
+ # Works the same as multi delegator, but randomly chooses an output to send each message.
91
+ balancer_logger = LogStashLogger.new(
92
+ type: :balancer,
93
+ outputs: [
94
+ { type: :udp, host: 'host1', port: 5228 },
95
+ { type: :udp, host: 'host2', port: 5228 }
96
+ ])
97
+
98
+ # Send messages to multiple loggers.
99
+ # Use this if you need to send different formats to different outputs.
100
+ # If you need to log to syslog, you must use this.
101
+ multi_logger = LogStashLogger.new(
102
+ type: :multi_logger,
103
+ outputs: [
104
+ { type: :file, path: 'log/development.log', formatter: ::Logger::Formatter },
105
+ { type: :tcp, host: 'localhost', port: 5228, formatter: :json }
106
+ ])
107
+
108
+ # The following messages are written to UDP port 5228:
109
+
110
+ logger.info 'test'
111
+ # {"message":"test","@timestamp":"2014-05-22T09:37:19.204-07:00","@version":"1","severity":"INFO","host":"[hostname]"}
112
+
113
+ logger.error '{"message": "error"}'
114
+ # {"message":"error","@timestamp":"2014-05-22T10:10:55.877-07:00","@version":"1","severity":"ERROR","host":"[hostname]"}
115
+
116
+ logger.debug message: 'test', foo: 'bar'
117
+ # {"message":"test","foo":"bar","@timestamp":"2014-05-22T09:43:24.004-07:00","@version":"1","severity":"DEBUG","host":"[hostname]"}
118
+
119
+ logger.warn LogStash::Event.new(message: 'test', foo: 'bar')
120
+ # {"message":"test","foo":"bar","@timestamp":"2014-05-22T16:44:37.364Z","@version":"1","severity":"WARN","host":"[hostname]"}
121
+
122
+ # Tagged logging
123
+ logger.tagged('foo') { logger.fatal('bar') }
124
+ # {"message":"bar","@timestamp":"2014-05-26T20:35:14.685-07:00","@version":"1","severity":"FATAL","host":"[hostname]","tags":["foo"]}
125
+ ```
126
+
127
+ ## URI Configuration
128
+ You can use a URI to configure your logstash logger instead of a hash. This is useful in environments
129
+ such as Heroku where you may want to read configuration values from the environment. The URI scheme
130
+ is `type://host:port/path?key=value`. Some sample URI configurations are given below.
131
+
132
+ ```
133
+ udp://localhost:5228
134
+ tcp://localhost:5229
135
+ unix:///tmp/socket
136
+ file:///path/to/file
137
+ redis://localhost:6379
138
+ kafka://localhost:9092
139
+ stdout:/
140
+ stderr:/
141
+ ```
142
+
143
+ Pass the URI into your logstash logger like so:
144
+
145
+ ```ruby
146
+ # Read the URI from an environment variable
147
+ logger = LogStashLogger.new(uri: ENV['LOGSTASH_URI'])
148
+ ```
149
+
150
+ ## Logstash Listener Configuration
151
+
152
+ In order for logstash to correctly receive and parse the events, you will need to
153
+ configure and run a listener that uses the `json_lines` codec. For example, to receive
154
+ events over UDP on port 5228:
155
+
156
+ ```ruby
157
+ input {
158
+ udp {
159
+ host => "0.0.0.0"
160
+ port => 5228
161
+ codec => json_lines
162
+ }
163
+ }
164
+ ```
165
+
166
+ File and Redis inputs should use the `json` codec instead. For more information
167
+ read the [Logstash docs](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json_lines.html).
168
+
169
+ See the [samples](https://github.com/dwbutler/logstash-logger/tree/master/samples) directory for more configuration samples.
170
+
171
+ ## SSL
172
+
173
+ If you are using TCP then there is the option of adding an SSL certificate to the options hash on initialize.
174
+
175
+ ```ruby
176
+ LogStashLogger.new(type: :tcp, port: 5228, ssl_certificate: "/path/to/certificate.crt")
177
+ ```
178
+
179
+ The SSL certificate and key can be generated using
180
+
181
+ openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout logstash.key -out logstash.crt
182
+
183
+ You can also enable SSL without a certificate:
184
+
185
+ ```ruby
186
+ LogStashLogger.new(type: :tcp, port: 5228, ssl_enable: true)
187
+ ```
188
+
189
+ Specify an SSL context to have more control over the behavior. For example,
190
+ set the verify mode:
191
+
192
+ ```ruby
193
+ ctx = OpenSSL::SSL::SSLContext.new
194
+ ctx.set_params(verify_mode: OpenSSL::SSL::VERIFY_NONE)
195
+ LogStashLogger.new(type: :tcp, port: 5228, ssl_context: ctx)
196
+ ```
197
+
198
+ The following Logstash configuration is required for SSL:
199
+
200
+ ```ruby
201
+ input {
202
+ tcp {
203
+ host => "0.0.0.0"
204
+ port => 5228
205
+ codec => json_lines
206
+ ssl_enable => true
207
+ ssl_cert => "/path/to/certificate.crt"
208
+ ssl_key => "/path/to/key.key"
209
+ }
210
+ }
211
+ ```
212
+
213
+ ### Hostname Verification
214
+
215
+ Hostname verification is enabled by default. Without further configuration,
216
+ the hostname supplied to `:host` will be used to verify the server's certificate
217
+ identity.
218
+
219
+ If you don't pass an `:ssl_context` or pass a falsey value to the
220
+ `:verify_hostname` option, hostname verification will not occur.
221
+
222
+ #### Examples
223
+
224
+ **Verify the hostname from the `:host` option**
225
+
226
+ ```ruby
227
+ ctx = OpenSSL::SSL::SSLContext.new
228
+ ctx.cert = '/path/to/cert.pem'
229
+ ctx.verify_mode = OpenSSL::SSL::VERIFY_PEER
230
+
231
+ LogStashLogger.new \
232
+ type: :tcp,
233
+ host: 'logstash.example.com'
234
+ port: 5228,
235
+ ssl_context: ctx
236
+ ```
237
+
238
+ **Verify a hostname different from the `:host` option**
239
+
240
+ ```ruby
241
+ LogStashLogger.new \
242
+ type: :tcp,
243
+ host: '1.2.3.4'
244
+ port: 5228,
245
+ ssl_context: ctx,
246
+ verify_hostname: 'server.example.com'
247
+ ```
248
+
249
+ **Explicitly disable hostname verification**
250
+
251
+ ```ruby
252
+ LogStashLogger.new \
253
+ type: :tcp,
254
+ host: '1.2.3.4'
255
+ port: 5228,
256
+ ssl_context: ctx,
257
+ verify_hostname: false
258
+ ```
259
+
260
+ ## Custom Log Fields
261
+
262
+ `LogStashLogger` by default will log a JSON object with the format below.
263
+
264
+ ```json
265
+ {
266
+ "message":"Some Message",
267
+ "@timestamp":"2015-01-29T10:43:32.196-05:00",
268
+ "@version":"1",
269
+ "severity":"INFO",
270
+ "host":"hostname"
271
+ }
272
+ ```
273
+
274
+ Some applications may need to attach additional metadata to each message.
275
+ The `LogStash::Event` can be manipulated directly by specifying a `customize_event` block in the `LogStashLogger` configuration.
276
+
277
+ ```ruby
278
+ config = LogStashLogger.configure do |config|
279
+ config.customize_event do |event|
280
+ event["other_field"] = "some_other_value"
281
+ end
282
+ end
283
+ ```
284
+
285
+ This configuration would result in the following output.
286
+
287
+ ```json
288
+ {
289
+ "message": "Some Message",
290
+ "@timestamp": "2015-01-29T10:43:32.196-05:00",
291
+ "@version": "1",
292
+ "severity": "INFO",
293
+ "host": "hostname",
294
+ "other_field": "some_other_value"
295
+ }
296
+ ```
297
+
298
+ This block has full access to the event, so you can remove fields, modify
299
+ existing fields, etc. For example, to remove the default timestamp:
300
+
301
+ ```ruby
302
+ config = LogStashLogger.configure do |config|
303
+ config.customize_event do |event|
304
+ event.remove('@timestamp')
305
+ end
306
+ end
307
+ ```
308
+
309
+ You can also customize events on a per-logger basis by passing a callable object
310
+ (lambda or proc) to the `customize_event` option when creating a logger:
311
+
312
+ ```ruby
313
+ LogStashLogger.new(customize_event: ->(event){ event['other_field'] = 'other_field' })
314
+ ```
315
+
316
+ ## Buffering / Automatic Retries
317
+
318
+ For devices that establish a connection to a remote service, log messages are buffered internally
319
+ and flushed in a background thread. If there is a connection problem, the
320
+ messages are held in the buffer and automatically resent until it is successful.
321
+ Outputs that support batch writing (Redis and Kafka) will write log messages in bulk from the
322
+ buffer. This functionality is implemented using a fork of
323
+ [Stud::Buffer](https://github.com/jordansissel/ruby-stud/blob/master/lib/stud/buffer.rb).
324
+ You can configure its behavior by passing the following options to LogStashLogger:
325
+
326
+ * :buffer_max_items - Max number of items to buffer before flushing. Defaults to 50.
327
+ * :buffer_max_interval - Max number of seconds to wait between flushes. Defaults to 5.
328
+ * :drop_messages_on_flush_error - Drop messages when there is a flush error. Defaults to false.
329
+ * :drop_messages_on_full_buffer - Drop messages when the buffer is full. Defaults to true.
330
+ * :sync - Flush buffer every time a message is received (blocking). Defaults to false.
331
+ * :buffer_flush_at_exit - Flush messages when exiting the program. Defaults to true.
332
+ * :buffer_logger - Logger to write buffer debug/error messages to. Defaults to none.
333
+
334
+ You can turn buffering off by setting `sync = true`.
335
+
336
+ Please be aware of the following caveats to this behavior:
337
+
338
+ * It's possible for duplicate log messages to be sent when retrying. For outputs like Redis and
339
+ Kafka that write in batches, the whole batch could get re-sent. If this is a problem, you
340
+ can add a UUID field to each event to uniquely identify it. You can either do this
341
+ in a `customize_event` block, or by using logstash's
342
+ [UUID filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-uuid.html).
343
+ * It's still possible to lose log messages. Ruby won't detect a TCP/UDP connection problem
344
+ immediately. In my testing, it took Ruby about 4 seconds to notice the receiving end was down
345
+ and start raising exceptions. Since logstash listeners over TCP/UDP do not acknowledge received
346
+ messages, it's not possible to know which log messages to re-send.
347
+ * When `sync` is turned off, Ruby may buffer data internally before writing to
348
+ the IO device. This is why you may not see messages written immediately to a
349
+ UDP or TCP socket, even though LogStashLogger's buffer is periodically flushed.
350
+
351
+ ## Full Buffer
352
+
353
+ By default, messages are discarded when the buffer gets full. This can happen
354
+ if the output source is down for too long or log messages are being received
355
+ too quickly. If your application suddenly terminates (for example, by SIGKILL or a power outage),
356
+ the whole buffer will be lost.
357
+
358
+ You can make message loss less likely by increasing `buffer_max_items`
359
+ (so that more events can be held in the buffer), and decreasing `buffer_max_interval` (to wait
360
+ less time between flushes). This will increase memory pressure on your application as log messages
361
+ accumulate in the buffer, so make sure you have allocated enough memory to your process.
362
+
363
+ If you don't want to lose messages when the buffer gets full, you can set
364
+ `drop_messages_on_full_buffer = false`. Note that if the buffer gets full, any
365
+ incoming log message will block, which could be undesirable.
366
+
367
+ ## Sync Mode
368
+
369
+ All logger outputs support a `sync` setting. This is analogous to the "sync mode" setting on Ruby IO
370
+ objects. When set to `true`, output is immediately flushed and is not buffered internally. Normally,
371
+ for devices that connect to a remote service, buffering is a good thing because
372
+ it improves performance and reduces the likelihood of errors affecting the program. For these devices,
373
+ `sync` defaults to `false`, and it is recommended to leave the default value.
374
+ You may want to turn sync mode on for testing, for example if you want to see
375
+ log messages immediately after they are written.
376
+
377
+ It is recommended to turn sync mode on for file and Unix socket outputs. This
378
+ ensures that log messages from different threads or proceses are written correctly on separate lines.
379
+
380
+ See [#44](https://github.com/dwbutler/logstash-logger/issues/44) for more details.
381
+
382
+ ## Error Handling
383
+
384
+ If an exception occurs while writing a message to the device, the exception is
385
+ logged using an internal logger. By default, this logs to $stderr. You can
386
+ change the error logger by setting `LogStashLogger.configuration.default_error_logger`, or by passsing
387
+ your own logger object in the `:error_logger` configuration key when
388
+ instantiating a LogStashLogger.
389
+
390
+ ## Logger Silencing
391
+
392
+ LogStashLogger provides support for Rails-style logger silencing. The
393
+ implementation was extracted from Rails, but has no dependencies, so it can be
394
+ used outside of a Rails app. The interface is the same as in Rails:
395
+
396
+ ```ruby
397
+ logger.silence(temporary_level) do
398
+ ...
399
+ end
400
+ ```
401
+
402
+ ## Custom Logger Class
403
+
404
+ By default, LogStashLogger creates a logger that extends Ruby's built in `Logger` class.
405
+ If you require a different logger implementation, you can use a different class
406
+ by passing in the class with the `logger_class` option.
407
+
408
+ Note that for syslog, the `Syslog::Logger` class is required and cannot be
409
+ changed.
410
+
411
+ ## Rails Integration
412
+
413
+ Supports Rails 4.2 and 5.x.
414
+
415
+ By default, every Rails log message will be written to logstash in `LogStash::Event` JSON format.
416
+
417
+ For minimal, more-structured logstash events, try one of the following gems:
418
+
419
+ * [lograge](https://github.com/roidrage/lograge)
420
+ * [yarder](https://github.com/rurounijones/yarder)
421
+
422
+ Currently these gems output a JSON string, which LogStashLogger then parses.
423
+ Future versions of these gems could potentially have deeper integration with LogStashLogger
424
+ (e.g. by directly writing `LogStash::Event` objects).
425
+
426
+ ### Rails Configuration
427
+
428
+ Add the following to your `config/environments/production.rb`:
429
+
430
+ #### Common Options
431
+
432
+ ```ruby
433
+ # Optional, Rails sets the default to :info
434
+ config.log_level = :debug
435
+
436
+ # Optional, Rails 4 defaults to true in development and false in production
437
+ config.autoflush_log = true
438
+
439
+ # Optional, use a URI to configure. Useful on Heroku
440
+ config.logstash.uri = ENV['LOGSTASH_URI']
441
+
442
+ # Optional. Defaults to :json_lines. If there are multiple outputs,
443
+ # they will all share the same formatter.
444
+ config.logstash.formatter = :json_lines
445
+
446
+ # Optional, the logger to log writing errors to. Defaults to logging to $stderr
447
+ config.logstash.error_logger = Logger.new($stderr)
448
+
449
+ # Optional, max number of items to buffer before flushing. Defaults to 50
450
+ config.logstash.buffer_max_items = 50
451
+
452
+ # Optional, max number of seconds to wait between flushes. Defaults to 5
453
+ config.logstash.buffer_max_interval = 5
454
+
455
+ # Optional, drop message when a connection error occurs. Defaults to false
456
+ config.logstash.drop_messages_on_flush_error = false
457
+
458
+ # Optional, drop messages when the buffer is full. Defaults to true
459
+ config.logstash.drop_messages_on_full_buffer = true
460
+ ```
461
+
462
+ #### UDP
463
+ ```ruby
464
+ # Optional, defaults to '0.0.0.0'
465
+ config.logstash.host = 'localhost'
466
+
467
+ # Optional, defaults to :udp.
468
+ config.logstash.type = :udp
469
+
470
+ # Required, the port to connect to
471
+ config.logstash.port = 5228
472
+ ```
473
+
474
+ #### TCP
475
+
476
+ ```ruby
477
+ # Optional, defaults to '0.0.0.0'
478
+ config.logstash.host = 'localhost'
479
+
480
+ # Required, the port to connect to
481
+ config.logstash.port = 5228
482
+
483
+ # Required
484
+ config.logstash.type = :tcp
485
+
486
+ # Optional, enables SSL
487
+ config.logstash.ssl_enable = true
488
+ ```
489
+
490
+ #### Unix Socket
491
+
492
+ ```ruby
493
+ # Required
494
+ config.logstash.type = :unix
495
+
496
+ # Required
497
+ config.logstash.path = '/tmp/sock'
498
+ ```
499
+
500
+ #### Syslog
501
+
502
+ If you're on Ruby 1.9, add `Syslog::Logger` v2 to your Gemfile:
503
+
504
+ gem 'SyslogLogger', '2.0'
505
+
506
+ If you're on Ruby 2+, `Syslog::Logger` is already built into the standard library.
507
+
508
+ ```ruby
509
+ # Required
510
+ config.logstash.type = :syslog
511
+
512
+ # Optional. Defaults to 'ruby'
513
+ config.logstash.program_name = 'MyApp'
514
+
515
+ # Optional default facility level. Only works in Ruby 2+
516
+ config.logstash.facility = Syslog::LOG_LOCAL0
517
+ ```
518
+
519
+ #### Redis
520
+
521
+ Add the redis gem to your Gemfile:
522
+
523
+ gem 'redis'
524
+
525
+ ```ruby
526
+ # Required
527
+ config.logstash.type = :redis
528
+
529
+ # Optional, will default to the 'logstash' list
530
+ config.logstash.list = 'logstash'
531
+
532
+ # All other options are passed in to the Redis client
533
+ # Supported options include host, port, path, password, url
534
+ # Example:
535
+
536
+ # Optional, Redis will default to localhost
537
+ config.logstash.host = 'localhost'
538
+
539
+ # Optional, Redis will default to port 6379
540
+ config.logstash.port = 6379
541
+ ```
542
+
543
+ #### Kafka
544
+
545
+ Add the poseidon gem to your Gemfile:
546
+
547
+ gem 'poseidon'
548
+
549
+ ```ruby
550
+ # Required
551
+ config.logstash.type = :kafka
552
+
553
+ # Optional, will default to the 'logstash' topic
554
+ config.logstash.path = 'logstash'
555
+
556
+ # Optional, will default to the 'logstash-logger' producer
557
+ config.logstash.producer = 'logstash-logger'
558
+
559
+ # Optional, will default to localhost:9092 host/port
560
+ config.logstash.hosts = ['localhost:9092']
561
+
562
+ # Optional, will default to 1s backoff
563
+ config.logstash.backoff = 1
564
+
565
+ ```
566
+
567
+ #### Kinesis
568
+
569
+ Add the aws-sdk gem to your Gemfile:
570
+
571
+ # aws-sdk >= 3.0
572
+ gem 'aws-sdk-kinesis'
573
+
574
+ # aws-sdk < 3.0
575
+ gem 'aws-sdk'
576
+
577
+ ```ruby
578
+ # Required
579
+ config.logstash.type = :kinesis
580
+
581
+ # Optional, will default to the 'logstash' stream
582
+ config.logstash.stream = 'my-stream-name'
583
+
584
+ # Optional, will default to 'us-east-1'
585
+ config.logstash.aws_region = 'us-west-2'
586
+
587
+ # Optional, will default to the AWS_ACCESS_KEY_ID environment variable
588
+ config.logstash.aws_access_key_id = 'ASKASKHLD12341'
589
+
590
+ # Optional, will default to the AWS_SECRET_ACCESS_KEY environment variable
591
+ config.logstash.aws_secret_access_key = 'ASKASKHLD1234123412341234'
592
+
593
+ ```
594
+
595
+ #### Firehose
596
+
597
+ Add the aws-sdk gem to your Gemfile:
598
+
599
+ # aws-sdk >= 3.0
600
+ gem 'aws-sdk-firehose'
601
+
602
+ # aws-sdk < 3.0
603
+ gem 'aws-sdk'
604
+
605
+ ```ruby
606
+ # Required
607
+ config.logstash.type = :firehose
608
+
609
+ # Optional, will default to the 'logstash' delivery stream
610
+ config.logstash.stream = 'my-stream-name'
611
+
612
+ # Optional, will default to AWS default region config chain
613
+ config.logstash.aws_region = 'us-west-2'
614
+
615
+ # Optional, will default to AWS default credential provider chain
616
+ config.logstash.aws_access_key_id = 'ASKASKHLD12341'
617
+
618
+ # Optional, will default to AWS default credential provider chain
619
+ config.logstash.aws_secret_access_key = 'ASKASKHLD1234123412341234'
620
+
621
+ ```
622
+
623
+ #### File
624
+
625
+ ```ruby
626
+ # Required
627
+ config.logstash.type = :file
628
+
629
+ # Optional, defaults to Rails log path
630
+ config.logstash.path = 'log/production.log'
631
+ ```
632
+
633
+ #### IO
634
+
635
+ ```ruby
636
+ # Required
637
+ config.logstash.type = :io
638
+
639
+ # Required
640
+ config.logstash.io = io
641
+ ```
642
+
643
+ #### Multi Delegator
644
+
645
+ ```ruby
646
+ # Required
647
+ config.logstash.type = :multi_delegator
648
+
649
+ # Required
650
+ config.logstash.outputs = [
651
+ {
652
+ type: :file,
653
+ path: 'log/production.log'
654
+ },
655
+ {
656
+ type: :udp,
657
+ port: 5228,
658
+ host: 'localhost'
659
+ }
660
+ ]
661
+ ```
662
+
663
+ #### Multi Logger
664
+
665
+ ```ruby
666
+ # Required
667
+ config.logstash.type = :multi_logger
668
+
669
+ # Required. Each logger may have its own formatter.
670
+ config.logstash.outputs = [
671
+ {
672
+ type: :file,
673
+ path: 'log/production.log',
674
+ formatter: ::Logger::Formatter
675
+ },
676
+ {
677
+ type: :udp,
678
+ port: 5228,
679
+ host: 'localhost'
680
+ }
681
+ ]
682
+ ```
683
+
684
+ ### Logging HTTP request data
685
+
686
+ In web applications, you can log data from HTTP requests (such as headers) using the
687
+ [RequestStore](https://github.com/steveklabnik/request_store) middleware. The following
688
+ example assumes Rails.
689
+
690
+ ```ruby
691
+ # in Gemfile
692
+ gem 'request_store'
693
+ ```
694
+
695
+ ```ruby
696
+ # in application.rb
697
+ LogStashLogger.configure do |config|
698
+ config.customize_event do |event|
699
+ event["session_id"] = RequestStore.store[:load_balancer_session_id]
700
+ end
701
+ end
702
+ ```
703
+
704
+ ```ruby
705
+ # in app/controllers/application_controller.rb
706
+ before_filter :track_load_balancer_session_id
707
+
708
+ def track_load_balancer_session_id
709
+ RequestStore.store[:load_balancer_session_id] = request.headers["X-LOADBALANCER-SESSIONID"]
710
+ end
711
+ ```
712
+
713
+ ## Cleaning up resources when forking
714
+
715
+ If your application forks (as is common with many web servers) you will need to
716
+ manage cleaning up resources on your LogStashLogger instances. The instance method
717
+ `#reset` is available for this purpose. Here is sample configuration for
718
+ several common web servers used with Rails:
719
+
720
+ Passenger:
721
+ ```ruby
722
+ ::PhusionPassenger.on_event(:starting_worker_process) do |forked|
723
+ Rails.logger.reset
724
+ end
725
+ ```
726
+
727
+ Puma:
728
+ ```ruby
729
+ # In config/puma.rb
730
+ on_worker_boot do
731
+ Rails.logger.reset
732
+ end
733
+ ```
734
+
735
+ Unicorn
736
+ ```ruby
737
+ # In config/unicorn.rb
738
+ after_fork do |server, worker|
739
+ Rails.logger.reset
740
+ end
741
+ ```
742
+
743
+ ## Ruby Compatibility
744
+
745
+ Verified to work with:
746
+
747
+ * MRI Ruby 2.2 - 2.5
748
+ * JRuby 9.x
749
+ * Rubinius
750
+
751
+ Ruby versions < 2.2 are EOL'ed and no longer supported.
752
+
753
+ ## What type of logger should I use?
754
+
755
+ It depends on your specific needs, but most applications should use the default (UDP). Here are the advantages and
756
+ disadvantages of each type:
757
+
758
+ * UDP is faster than TCP because it's asynchronous (fire-and-forget). However, this means that log messages could get dropped.
759
+ This is okay for many applications.
760
+ * TCP verifies that every message has been received via two-way communication. It also supports SSL for secure transmission
761
+ of log messages over a network. This could slow your app down to a crawl if the TCP listener is under heavy load.
762
+ * A file is simple to use, but you will have to worry about log rotation and running out of disk space.
763
+ * Writing to a Unix socket is faster than writing to a TCP or UDP port, but only works locally.
764
+ * Writing to Redis is good for distributed setups that generate tons of logs. However, you will have another moving part and
765
+ have to worry about Redis running out of memory.
766
+ * Writing to stdout is only recommended for debugging purposes.
767
+
768
+ For a more detailed discussion of UDP vs TCP, I recommend reading this article:
769
+ [UDP vs. TCP](http://gafferongames.com/networking-for-game-programmers/udp-vs-tcp/)
770
+
771
+ ## Troubleshooting
772
+
773
+ ### Logstash never receives any logs
774
+ If you are using a device backed by a Ruby IO object (such as a file, UDP socket, or TCP socket), please be aware that Ruby
775
+ keeps its own internal buffer. Despite the fact that LogStashLogger buffers
776
+ messages and flushes them periodically, the data written to the IO object can
777
+ be buffered by Ruby internally indefinitely, and may not even write until the
778
+ program terminates. If this bothers you or you need to see log messages
779
+ immediately, your only recourse is to set the `sync: true` option.
780
+
781
+ ### JSON::GeneratorError
782
+ Your application is probably attempting to log data that is not encoded in a valid way. When this happens, Ruby's
783
+ standard JSON library will raise an exception. You may be able to overcome this by swapping out a different JSON encoder
784
+ such as Oj. Use the [oj_mimic_json](https://github.com/ohler55/oj_mimic_json) gem to use Oj for JSON generation.
785
+
786
+ ### No logs getting sent on Heroku
787
+ Heroku recommends installing the [rails_12factor](https://github.com/heroku/rails_12factor) so that logs get sent to STDOUT.
788
+ Unfortunately, this overrides LogStashLogger, preventing logs from being sent to their configured destination. The solution
789
+ is to remove `rails_12factor` from your Gemfile.
790
+
791
+ ### Logging eventually stops in production
792
+ This is most likely not a problem with LogStashLogger, but rather a different gem changing the log level of `Rails.logger`.
793
+ This is especially likely if you're using a threaded server such as Puma, since gems often change the log level of
794
+ `Rails.logger` in a non thread-safe way. See [#17](https://github.com/dwbutler/logstash-logger/issues/17) for more information.
795
+
796
+ ### Sometimes two lines of JSON log messages get sent as one message
797
+ If you're using UDP output and writing to a logstash listener, you are most likely encountering a bug in the UDP implementation
798
+ of the logstash listener. There is no known fix at this time. See [#43](https://github.com/dwbutler/logstash-logger/issues/43)
799
+ for more information.
800
+
801
+ ### Errno::EMSGSIZE - Message too long
802
+ A known drawback of using TCP or UDP is the 65535 byte limit on total message size. To workaround
803
+ this issue, you will have to truncate the message by setting the max message size:
804
+
805
+ ```ruby
806
+ LogStashLogger.configure do |config|
807
+ config.max_message_size = 2000
808
+ end
809
+ ```
810
+
811
+ This will truncate only the `message` field of the LogStash Event. So make sure
812
+ you set the max message size significantly less than 65535 bytes to make room
813
+ for other fields.
814
+
815
+ ## Breaking changes
816
+
817
+ ### Version 0.25+
818
+
819
+ Rails 3.2, MRI Ruby < 2.2, and JRuby 1.7 are no longer supported, since they have been
820
+ EOL'ed. If you are on an older version of Ruby, you will need to use 0.24 or below.
821
+
822
+ ### Version 0.5+
823
+ * The `source` event key has been replaced with `host` to better match the latest logstash.
824
+ * The `(host, port, type)` constructor has been deprecated in favor of an options hash constructor.
825
+
826
+ ### Version 0.4+
827
+ `LogStash::Event` uses the v1 format starting version 1.2+. If you're using the v1, you'll need to install
828
+ LogStashLogger version 0.4+. This is not backwards compatible with the old `LogStash::Event` v1.1.5, which uses
829
+ the v0 format.
830
+
831
+ ### Version 0.3+
832
+ Earlier versions of this gem (<= 0.2.1) only implemented a TCP connection.
833
+ Newer versions (>= 0.3) also implement UDP, and use that as the new default.
834
+ Please be aware if you are using the default constructor and still require TCP, you should add an additional argument:
835
+
836
+ ```ruby
837
+ # Now defaults to UDP instead of TCP
838
+ logger = LogStashLogger.new('localhost', 5228)
839
+ # Explicitly specify TCP instead of UDP
840
+ logger = LogStashLogger.new('localhost', 5228, :tcp)
841
+ ```
842
+
843
+ ## Contributors
844
+ * [David Butler](https://github.com/dwbutler)
845
+ * [pctj101](https://github.com/pctj101)
846
+ * [Gary Rennie](https://github.com/Gazler)
847
+ * [Nick Ethier](https://github.com/nickethier)
848
+ * [Arron Mabrey](https://github.com/arronmabrey)
849
+ * [Jan Schulte](https://github.com/schultyy)
850
+ * [Kurt Preston](https://github.com/KurtPreston)
851
+ * [Chris Blatchley](https://github.com/chrisblatchley)
852
+ * [Felix Bechstein](https://github.com/felixb)
853
+ * [Vadim Kazakov](https://github.com/yads)
854
+ * [Anil Rhemtulla](https://github.com/AnilRh)
855
+ * [Nikita Vorobei](https://github.com/Nikita-V)
856
+ * [fireboy1919](https://github.com/fireboy1919)
857
+ * [Mike Gunderloy](https://github.com/ffmike)
858
+ * [Vitaly Gorodetsky](https://github.com/vitalis)
859
+ * [Courtland Caldwell](https://github.com/caldwecr)
860
+ * [Bibek Shrestha](https://github.com/bibstha)
861
+ * [Alex Ianus](https://github.com/aianus)
862
+ * [Craig Read](https://github.com/Catharz)
863
+ * [glaszig](https://github.com/glaszig)
864
+ * [Bin Lan](https://github.com/lanxx019)
865
+ * [Joao Fernandes](https://github.com/jcmfernandes)
866
+ * [CoolElvis](https://github.com/coolelvis)
867
+ * [Sergey Pyankov](https://github.com/esergion)
868
+ * [Alec Hoey](https://github.com/alechoey)
869
+ * [Alexey Krasnoperov](https://github.com/AlexeyKrasnoperov)
870
+ * [Gabriel de Oliveira](https://github.com/gdeoliveira)
871
+ * [Vladislav Syabruk](https://github.com/SeTeM)
872
+ * [Matus Vacula](https://github.com/matus-vacula)
873
+
874
+ ## Contributing
875
+
876
+ 1. Fork it
877
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
878
+ 3. Commit your changes (`git commit -am 'Add some feature'`)
879
+ 4. Push to the branch (`git push origin my-new-feature`)
880
+ 5. Create new Pull Request