semantic_logger 2.8.0 → 2.9.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 51a1e44ec07ae96cd5dad5b3a3dc2c6377960b15
4
- data.tar.gz: 820304cb8615f71b1f921869fcc70e0766ef7258
3
+ metadata.gz: 05fe9c7ccd2177383e5f4a8cf4b5b6d0bf84df71
4
+ data.tar.gz: 0516f2bb270d5ea5d78b67217fd02745ebe4c3ce
5
5
  SHA512:
6
- metadata.gz: 3faa5b434bc505b9d2045f7f00c91feae01f89bcd3a498ea7e0fe8f831d75fd276f05fd059ddd149ba2755d3ab6dfa3abeddc6d9c025191e75a4517c304c155f
7
- data.tar.gz: d0691ea0ad358cf91c266e5e9005d510aed493f4adb8255423a425133c7258248d53482420e06623db1009a620adcd50ec616c56ad687757a4427eabbc16c603
6
+ metadata.gz: 39f2f6909cdd7e1623fd099226849a2086873b52334ddabbde84067a12b3e88d1058384b346f09c11f1c3c357b38758ae6e51dd649e3d39f2765304f40ac59d5
7
+ data.tar.gz: 025cca0bde8b1733a023ba9139b474d9f14e85444b518e24764b2a346aa8676be5441e4e006866caf8a103027156e4c39f8b30af3cfd144d93a2fc96f26840ed
data/LICENSE.txt CHANGED
@@ -186,7 +186,7 @@
186
186
  same "printed page" as the copyright notice for easier
187
187
  identification within third-party archives.
188
188
 
189
- Copyright 2012, 2013 Reid Morrison
189
+ Copyright 2012, 2013, 2014 Reid Morrison
190
190
 
191
191
  Licensed under the Apache License, Version 2.0 (the "License");
192
192
  you may not use this file except in compliance with the License.
data/README.md CHANGED
@@ -1,1037 +1,13 @@
1
1
  semantic_logger [![Build Status](https://secure.travis-ci.org/reidmorrison/semantic_logger.png?branch=master)](http://travis-ci.org/reidmorrison/semantic_logger)
2
2
  ===============
3
3
 
4
- Scalable, next generation logging for Ruby
4
+ Next generation logging system for Ruby to support highly concurrent, high throughput, low latency systems
5
5
 
6
6
  * http://github.com/reidmorrison/semantic_logger
7
7
 
8
- ## Overview
8
+ ## Documentation
9
9
 
10
- Semantic Logger takes logging in Ruby to the next level by adding several new
11
- capabilities to the commonly used Logging API:
12
-
13
- High Performance
14
-
15
- * Logging is performed in a separate thread so as not to slow down the application
16
- whilst logging to one or more destinations
17
- * Supports logging from hundreds of concurrent threads in the same process
18
-
19
- Drop-in Replacement
20
-
21
- * Simple drop-in replacement for the Ruby, or the Rails loggers
22
- * Supports current common logging interface
23
- * No changes to existing to code to use new logger ( other than replacing the logger )
24
-
25
- Dynamic
26
-
27
- * Increase the log level at runtime for just one class
28
- * For example enable debug level logging for a single class (logging instance)
29
- while the program is running to get more detailed logging in production for just that class
30
- * Change the default global logging level for all classes, unless that class has
31
- specifically been overridden above
32
- * Use UNIX signals to change the log level for a running process
33
-
34
- Tagged Logging
35
-
36
- * Supply custom data to be added to every log entry within a block of code,
37
- including libraries and existing gems
38
- * Tagged logging is critical for any high traffic site so that one can narrow
39
- down log entries for a single call that is mixed in with log entries
40
- from hundreds of other log entries
41
-
42
- Payload support
43
-
44
- * Aside from the regular log message, a hash payload can also be supplied with
45
- every log entry
46
- * Very powerful when logging to NOSQL destinations that allow queries against
47
- any data in the payload
48
-
49
- Exceptions
50
-
51
- * Directly log exceptions
52
- * Semantic Logger standardizes the logging of exceptions with their backtraces
53
- to text destinations and writes the exception elements as a hash to NOSQL
54
- destinations
55
-
56
- Benchmarking
57
-
58
- * The performance of any block of code can be measured and logged at the same time
59
- depending on the active log level
60
- * Supports only logging when the block of code exceeds a specified number of milli-seconds.
61
- Makes it easy to find bottlenecks when the system suddenly slows down in production
62
- * Exceptions thrown in the block of code can also be logged so as to aid in finding
63
- exceptions that may be discarded or hidden by the application
64
- * Benchmarked data can also be forwarded to external metric systems so that the
65
- performance of these blocks can be measured and/or monitored over time
66
-
67
- Thread Safe
68
-
69
- * Semantic Logger is completely thread safe and all methods can be called
70
- concurrently from any thread
71
- * Tagged logging keeps any tagging data on a per-thread basis to ensure that
72
- tags from different threads are not inter-mingled
73
- * Supports highly concurrent environments running hundreds of threads
74
- * Each appender writes all log entries sequentially in the appender thread so
75
- that log entries are written in the correct sequence
76
- * Avoids issues that other loggers experience when multiple threads try to write
77
- to the same log file at the same time creating partial and overwritten log
78
- entries in the log file
79
-
80
- Thread Aware
81
-
82
- * Includes the process id, and thread name or thread id in every log entry so that
83
- log entries from different processes and even threads are easily discernable
84
- * Human readable names can be assigned to every thread for logging purposes
85
-
86
- Trace Level
87
-
88
- * :trace is a new level common in other languages and is commonly used for
89
- logging trace level detail. It is intended for logging data at level below
90
- :debug.
91
- * :trace can be used for logging the actual data sent or received over the network
92
- that is rarely needed but is critical when things are not working as expected.
93
- * Since :trace can be enabled on a per class basis it can even be turned on
94
- in production to resolve what was actually sent to an external vendor
95
-
96
- Multiple Destinations
97
-
98
- * Log to multiple destinations at the same time ( File and MongoDB, etc.. )
99
- * Each destination can also have its own log level.
100
- For example, only log :info and above to MongoDB, or :warn and above to a
101
- second log file
102
-
103
- Semantic Capabilities
104
-
105
- * With Semantic Logger it is simple to mix-in additional semantic information with
106
- every log entry
107
- * The application or class name is automatically included for every log entry under
108
- a specific logging instance
109
- * Includes the duration of blocks of code
110
- * Any hash containing context specific information such as user_id or location information
111
-
112
- Beyond Tagged Logging
113
-
114
- * Supply entire hash of custom data to be added to the payload of every log entry
115
- within a block of code, including libraries and existing gems
116
-
117
- NOSQL Destinations
118
-
119
- * Every log entry is broken down into elements that NOSQL data stores can understand:
120
-
121
- ```json
122
- {
123
- "_id" : ObjectId("5034fa48e3f3fea945e83ef2"),
124
- "time" : ISODate("2012-08-22T15:27:04.409Z"),
125
- "host_name" : "release",
126
- "pid" : 16112,
127
- "thread_name" : "main",
128
- "name" : "UserLocator",
129
- "level" : "debug",
130
- "message" : "Fetch user information",
131
- "duration" : 12,
132
- "payload" : {
133
- "user" : "Jack",
134
- "zip_code" : 12345,
135
- "location" : "US"
136
- }
137
- }
138
- ```
139
-
140
- Customizable
141
-
142
- * Custom formatting by destination
143
- * Easy to "roll your own" destination (Appender).
144
- For example to log to Hadoop, Redis, etc..
145
-
146
- ## Introduction
147
-
148
- Semantic Logger is a Logger that supports logging of meta-data, along with text messages
149
- to multiple appenders
150
-
151
- An appender is a Logging destination such as a File, MongoDB collection, etc..
152
- Multiple Appenders can be active at the same time. All log entries are written
153
- to each appender.
154
-
155
- Machines can understand the logged data without having to use
156
- complex Regular Expressions or other text parsing techniques
157
-
158
- Semantic Logger, sits on top of existing logger implementations and can also
159
- be used as a drop in replacement for existing Ruby loggers.
160
- This allows the existing logging to be replaced immediately with the
161
- Semantic Logger Appenders, and over time the calls can be replaced with ones
162
- that contain the necessary meta-data.
163
-
164
- Example of current calls:
165
-
166
- ```ruby
167
- logger.info("Queried users table in #{duration} ms, with a result code of #{result}")
168
- ```
169
-
170
- For a machine to find all queries for table 'users' that took longer than
171
- 100 ms, would require using a regular expression just to extract the table name
172
- and duration, then apply the necessary logic. It also assumes that the text
173
- is not changed and that matches will not be found when another log entry has
174
- similar text output.
175
-
176
- This can be changed over time to:
177
-
178
- ```ruby
179
- logger.info("Queried table",
180
- :duration => duration,
181
- :result => result,
182
- :table => "users",
183
- :action => "query")
184
- ```
185
-
186
- Using the MongoDB appender, we can easily find all queries for table 'users'
187
- that took longer than 100 ms:
188
-
189
- ```javascript
190
- db.logs.find({"payload.table":"users", "payload.action":"query", "payload.duration":{$gt:100} })
191
- ```
192
-
193
- Since Semantic Logger can call existing Loggers, it does not force end-users
194
- to have to adopt a Semantic aware adapter. Although, such adapters create
195
- tremendous value in the problem monitoring and determination processes.
196
-
197
- ## Logging API
198
-
199
- ### Standard Logging methods
200
-
201
- The Semantic Logger logging API supports the existing logging interface for
202
- the Rails and Ruby Loggers. For example:
203
-
204
- ```ruby
205
- logger.info("Hello World")
206
- ```
207
-
208
- Or to query whether a specific log level is set
209
-
210
- ```ruby
211
- logger.info?
212
- ```
213
-
214
- The following traditional logging methods are available
215
-
216
- ```ruby
217
- logger.trace("Low level trace information such as data sent over a socket")
218
- logger.debug("Debugging information to aid with problem determination")
219
- logger.info("Informational message such as request received")
220
- logger.warn("Warn about something in the system")
221
- logger.error("An error occurred during processing")
222
- logger.fatal("Oh no something really bad happened")
223
- ```
224
-
225
- Each of the above calls can take additional parameters, for example:
226
-
227
- ```ruby
228
- log.info(message, payload=nil, exception=nil, &block)
229
- ```
230
-
231
- Parameters
232
-
233
- - message: The text message to log.
234
- Mandatory only if no block is supplied
235
- - payload: Optional, either a Ruby Exception object or a Hash
236
- - exception: Optional, Ruby Exception object. Allows both an exception and a payload to be logged
237
- - block: The optional block is executed only if the corresponding log level
238
- is active. Can be used to prevent unnecessary calculations of debug data in
239
- production.
240
-
241
- Examples:
242
-
243
- ```ruby
244
- logger.debug("Calling Supplier")
245
-
246
- logger.debug("Calling Supplier", :request => 'update', :user => 'Jack')
247
-
248
- logger.debug { "A total of #{result.inject(0) {|sum, i| i+sum }} were processed" }
249
- ```
250
-
251
- ## Exceptions
252
-
253
- The Semantic Logger adds an optional parameter to the existing log methods so that
254
- a corresponding Exception can be logged in a standard way
255
-
256
- ```ruby
257
- begin
258
- # ... Code that can raise an exception
259
- rescue Exception => exception
260
- logger.error("Oops external call failed", exception)
261
- # Re-raise or handle the exception
262
- raise exception
263
- end
264
- ```
265
-
266
- ### Payload
267
-
268
- The Semantic Logger adds an extra parameter to the existing log methods so that
269
- additional payload can be logged, such as a Hash or a Ruby Exception object.
270
-
271
- ```ruby
272
- logger.info("Oops external call failed", :result => :failed, :reason_code => -10)
273
- ```
274
-
275
- The additional payload is machine readable so that we don't have to write complex
276
- regular expressions so that a program can analyze log output. With the MongoDB
277
- appender the payload is written directly to MongoDB as part of the document and
278
- is therefore fully searchable
279
-
280
- ### Benchmarking
281
-
282
- Another common logging requirement is to measure the time it takes to execute a block
283
- of code based on the log level. For example:
284
-
285
- ```ruby
286
- Rails.logger.benchmark_info "Calling external interface" do
287
- # Code to call external service ...
288
- end
289
- ```
290
-
291
- The following output will be written to file:
292
-
293
- 2012-08-30 15:37:29.474 I [48308:ScriptThreadProcess: script/rails] (5.2ms) Rails -- Calling external interface
294
-
295
- If an exception is raised during the block the exception is logged
296
- at the same log level as the benchmark along with the duration and message.
297
- The exception will flow through to the caller unchanged
298
-
299
- The following benchmarking methods are available
300
-
301
- ```ruby
302
- logger.benchmark_trace("Low level trace information such as data sent over a socket")
303
- logger.benchmark_debug("Debugging information to aid with problem determination")
304
- logger.benchmark_info("Informational message such as request received")
305
- logger.benchmark_warn("Warn about something in the system")
306
- logger.benchmark_error("An error occurred during processing")
307
- logger.benchmark_fatal("Oh no something really bad happened")
308
- logger.benchmark(:info, "Informational message such as request received")
309
- ```
310
-
311
- Each of the above calls can take additional parameters, for example:
312
-
313
- ```ruby
314
- log.benchmark_info(message, params=nil) do
315
- # Measure how long it takes to run this block of code
316
- end
317
- ```
318
-
319
- Parameters
320
-
321
- - message: The mandatory text message to log.
322
- - params:
323
- ```
324
- :log_exception
325
- Control whether or how an exception thrown in the block is
326
- reported by Semantic Logger. Values:
327
- :full
328
- Log the exception class, message, and backtrace
329
- :partial
330
- Log the exception class and message
331
- The backtrace will not be logged
332
- :off
333
- Any unhandled exception raised in the block will not be logged
334
- Default: :partial
335
-
336
- :min_duration [Float]
337
- Only log if the block takes longer than this duration in ms
338
- Default: 0.0
339
-
340
- :payload [Hash]
341
- Optional, Hash payload
342
-
343
- :exception [Exception]
344
- Optional, Ruby Exception object to log along with the duration of the supplied block
345
-
346
- :duration [Float]
347
- Optional, supply the duration in ms that is logged when a block is not supplied
348
- If a block is not supplied then :duration is mandatory
349
- If a block is supplied :duration is ignored
350
-
351
- :metric [Object]
352
- Optional, when this parameter is supplied all subscribers will be notified of this
353
- metric, along with the Log Struct described below
354
- ```
355
-
356
- ### Logging levels
357
-
358
- The following logging levels are available through Semantic Logger
359
-
360
- :trace, :debug, :info, :warn, :error, :fatal
361
-
362
- The log levels are listed above in the order of precedence with the most detail to the least.
363
- For example :debug would include :info, :warn, :error, :fatal levels but not :trace
364
- And :fatal would only log :fatal error messages and nothing else
365
-
366
- :unknown has been mapped to :fatal for Rails and Ruby Logger
367
-
368
- :trace is a new level that is often used for tracing low level calls such
369
- as the data sent or received to external web services. It is also commonly used
370
- in the development environment for low level trace logging of methods calls etc.
371
-
372
- If only the rails logger is being used, then :trace level calls will be logged
373
- as debug calls only if the log level is set to trace
374
-
375
- ### Changing the Class name for Log Entries
376
-
377
- When Semantic Logger is included in a Rails project it automatically replaces the
378
- loggers for Rails, ActiveRecord::Base, ActionController::Base, and ActiveResource::Base
379
- with wrappers that set their Class name. For example in semantic_logger/railtie.rb:
380
-
381
- ```ruby
382
- ActiveRecord::Base.logger = SemanticLogger[ActiveRecord]
383
- ```
384
-
385
- By replacing their loggers we now get the class name in the text logging output:
386
-
387
- 2012-08-30 15:24:13.439 D [47900:main] ActiveRecord -- SQL (12.0ms) SELECT `schema_migrations`.`version` FROM `schema_migrations`
388
-
389
- It is recommended to include a class specific logger for all major classes that will
390
- be logging using the SemanticLogger::Loggable mix-in. For Example:
391
-
392
- ```ruby
393
- class ExternalSupplier
394
- # Lazy load logger class variable on first use
395
- include SemanticLogger::Loggable
396
-
397
- def call_supplier(amount, name)
398
- logger.debug "Calculating with amount", { :amount => amount, :name => name }
399
-
400
- # Measure and log on completion how long the call took to the external supplier
401
- logger.benchmark_info "Calling external interface" do
402
- # Code to call the external supplier ...
403
- end
404
- end
405
- end
406
- ```
407
-
408
- This will result in the log output identifying the log entry as from the ExternalSupplier class
409
-
410
- 2012-08-30 15:37:29.474 I [48308:ScriptThreadProcess: script/rails] (5.2ms) ExternalSupplier -- Calling external interface
411
-
412
- ### Changing the log level for a single class at runtime
413
-
414
- Since the logger is class specific, its log level can be changed dynamically at runtime.
415
- For example, to temporarily set the log level to :trace to diagnose an issue:
416
-
417
- ```ruby
418
- require 'semantic_logger'
419
-
420
- SemanticLogger.default_level = :info
421
- SemanticLogger.add_appender('example.log')
422
-
423
- class ExternalSupplier
424
- # Lazy load logger class variable on first use
425
- include SemanticLogger::Loggable
426
-
427
- def call_supplier(amount, name)
428
- logger.trace "Calculating with amount", { :amount => amount, :name => name }
429
-
430
- # Measure and log on completion how long the call took to the external supplier
431
- logger.benchmark_info "Calling external interface" do
432
- # Code to call the external supplier ...
433
- end
434
- end
435
- end
436
-
437
- # Create and use the class
438
- supplier = ExternalSupplier.new
439
- supplier.call_supplier(100, 'Jack')
440
-
441
- # Now change the log level to :trace
442
- ExternalSupplier.logger.level = :trace
443
-
444
- # Call the supplier, this time including trace level messages
445
- supplier.call_supplier(100, 'Jack')
446
-
447
- # Change the log level back to the default level
448
- ExternalSupplier.logger.level = SemanticLogger.default_level
449
- ```
450
-
451
- Below is the output from the above example showing the :trace log level message
452
- that was written during the second call to the ExternalSupplier:
453
-
454
- ```
455
- 2013-11-07 16:19:26.496 I [35674:main] (0.0ms) ExternalSupplier -- Calling external interface
456
-
457
- 2013-11-07 16:19:26.683 T [35674:main] ExternalSupplier -- Calculating with amount -- {:amount=>100, :name=>"Jack"}
458
- 2013-11-07 16:19:26.683 I [35674:main] (0.0ms) ExternalSupplier -- Calling external interface
459
- ```
460
-
461
- ### Change the global default logging level at runtime
462
-
463
- Log levels can be changed using signals on operating systems that support them.
464
- This allows log levels to be changed externally without requiring a restart
465
- of the running process.
466
-
467
- When the signal is raised, the global default log level rotates through the following
468
- log levels in the following order, starting from the current global default level:
469
-
470
- ```ruby
471
- :warn, :info, :debug, :trace
472
- ```
473
-
474
- If the current level is :trace it wraps around back to :warn
475
-
476
- Example:
477
-
478
- ```
479
- kill -SIGUSR2 1234
480
- ```
481
-
482
- #### Enabling Log Level Signal handler
483
-
484
- On startup SemanticLogger does not register any signals so that it does not
485
- interfere with any existing signal handlers. In order to enable the above log level
486
- changes the signal handler must be registered by calling `SemanticLogger.add_signal_handler`
487
-
488
- ```ruby
489
- require 'semantic_logger'
490
-
491
- # Enable signal handling for this process
492
- SemanticLogger.add_signal_handler('USR2')
493
-
494
- SemanticLogger.add_appender('development.log')
495
-
496
- logger = SemanticLogger['Example']
497
- logger.info "Hello World"
498
- ```
499
-
500
- Note: The changes to the logging level will not change for any classes where the
501
- log_level was set explicity within the application itself. The above signal only changes
502
- the global default level, which is used by loggers when their log level has not been changed.
503
-
504
- #### Change the log level without using signals
505
-
506
- If the application has another means of communicating without needing signals,
507
- the global default log level can be modified using `SemanticLogger.default_level=`
508
-
509
- ```ruby
510
- # Change the global default logging level for active loggers
511
- SemanticLogger.default_level = :debug
512
- ```
513
-
514
- ### Tagged Logging
515
-
516
- Semantic Logger allows any Ruby or Rails program to also include tagged logging.
517
-
518
- This means that any logging performed within a block, including any called
519
- libraries or gems to include the specified tag with every log entry.
520
-
521
- Using Tagged logging is critical in any highly concurrent environment so that
522
- one can quickly find all related log entries across all levels of code, and even
523
- across threads
524
-
525
- ```ruby
526
- logger.tagged(tracking_number) do
527
- logger.debug("Hello World")
528
- # ...
529
- end
530
- ```
531
-
532
- ### Beyond Tagged Logging
533
-
534
- Blocks of code can be tagged with not only values, but can be tagged with
535
- entire hashes of data. The additional hash of data will be merged into
536
- the payload of every log entry
537
-
538
- For example every corresponding log entry could include a hash containing
539
- a user_id, name, region, zip_code, tracking_number, etc...
540
-
541
- ```ruby
542
- logger.with_payload(:user => 'Jack', :zip_code => 12345) do
543
- logger.debug("Hello World")
544
- # ...
545
- end
546
- ```
547
-
548
- ### Named threads
549
-
550
- SemanticLogger logs the name or id of the thread in every log message.
551
-
552
- On Ruby MRI the thread name is by default the thread's object_id, For example: 70184354571980
553
-
554
- ```
555
- 2013-11-07 16:25:14.279627 I [35841:70184354571980] (0.0ms) ExternalSupplier -- Calling external interface
556
- ```
557
-
558
- To set a custom name for any thread so that it shows up in the logger:
559
-
560
- ```ruby
561
- Thread.current.name = "User calculation thread 32"
562
- ```
563
-
564
- Sample output:
565
-
566
- ```
567
- 2013-11-07 16:26:02.744139 I [35841:User calculation thread 32] (0.0ms) ExternalSupplier -- Calling external interface
568
- ```
569
-
570
- When running JRuby, Thread.current.name will also set the underlying thread name in the JVM
571
- which is very useful when monitoring the JVM via JMX using tools such as jconsole.
572
-
573
- #### NOTE:
574
-
575
- Make sure that the assigned thread name is unique otherwise it will be difficult
576
- to distinguish between concurrently running threads if they have the same name.
577
-
578
- For example, use the current thread object_id to ensure uniqueness:
579
-
580
- ```ruby
581
- Thread.current.name = "Worker Thread:#{Thread.current.object_id}"
582
- ```
583
-
584
- ### Metrics integration
585
-
586
- In production environments it is often necessary to not only measure the performance of a
587
- block of code using for example:
588
-
589
- ```ruby
590
- logger.benchmark_info "Calling external interface" do
591
- # Code to call the external supplier ...
592
- end
593
- ```
594
-
595
- A single subscriber can be defined to collect all the metrics and forward them
596
- for example to NewRelic:
597
-
598
- ```ruby
599
- # config/initializers/semantic_logger_metrics.rb
600
- SemanticLogger.on_metric do |log_struct|
601
- ::NewRelic::Agent.record_metric(log_struct.metric, log_struct.duration)
602
- end
603
- ```
604
-
605
- Add the :metric option to the log entry as follows:
606
-
607
- ```ruby
608
- logger.benchmark_info "Calling external interface", :metric => 'Custom/slow_action/beginning_work' do
609
- # Code to call the external supplier ...
610
- end
611
- ```
612
-
613
- ## Standalone SemanticLogger
614
-
615
- When using SemanticLogger inside of Rails all we need to do is include the
616
- rails_semantic_logger gem and the default Rails logger will be replaced with
617
- Semantic Logger.
618
-
619
- In a stand-alone or non-rails environment we can easily log to a file called
620
- 'development.log' as follows:
621
- ```ruby
622
- require 'semantic_logger'
623
- SemanticLogger.add_appender('development.log')
624
-
625
- logger = SemanticLogger['Example']
626
- logger.info "Hello World"
627
- ```
628
-
629
- By default it will only log :info and above, to log everything to the log file:
630
- ```ruby
631
- require 'semantic_logger'
632
- SemanticLogger.default_level = :trace
633
- SemanticLogger.add_appender('development.log')
634
-
635
- logger = SemanticLogger['Example']
636
- logger.info "Hello World"
637
- logger.trace "Low level trace information"
638
- ```
639
-
640
- By supplying multiple appenders Semantic Logger can write to multiple destinations
641
- at the same time. For example, log to a file and the screen:
642
- ```ruby
643
- require 'semantic_logger'
644
- SemanticLogger.default_level = :trace
645
- SemanticLogger.add_appender('development.log')
646
- SemanticLogger.add_appender(STDOUT)
647
-
648
- logger = SemanticLogger['Example']
649
- logger.info "Hello World"
650
- logger.trace "Low level trace information"
651
- ```
652
-
653
- To reduce the log level of logging to STDOUT to just :info and above, add the
654
- log_level such as :info as the second parameter when adding the appender:
655
- ```ruby
656
- require 'semantic_logger'
657
- SemanticLogger.default_level = :trace
658
- SemanticLogger.add_appender('development.log')
659
- SemanticLogger.add_appender(STDOUT, :info)
660
-
661
- logger = SemanticLogger['Example']
662
- logger.info "Hello World"
663
- logger.trace "Low level trace information"
664
- ```
665
-
666
- To log :debug and above to a log file, :error and above to $stderr, and :info
667
- and above to MongoDB:
668
- ```ruby
669
- require 'semantic_logger'
670
- require 'mongo'
671
-
672
- SemanticLogger.default_level = :debug
673
- SemanticLogger.add_appender('development.log')
674
- SemanticLogger.add_appender($stderr, :error)
675
-
676
- mongo_appender = SemanticLogger::Appender::MongoDB.new(
677
- :db => Mongodb::Connection.new['production_logging'],
678
- :collection_size => 25.gigabytes
679
- )
680
- SemanticLogger.add_appender(mongo_appender, :info)
681
-
682
- logger = SemanticLogger['Example']
683
- logger.info "Hello World"
684
- logger.trace "Low level trace information"
685
- logger.error "Oops an error occurred"
686
- logger.info("Login time", :user => 'Mary', :duration => 230, :ip_address=>'192.168.0.1')
687
- ```
688
-
689
- When starting out with Semantic Logger it can be useful to gain all the benefits
690
- of Semantic Logger and still continue to log to an existing logger:
691
- ```ruby
692
- require 'logger'
693
- require 'semantic_logger'
694
-
695
- # Built-in Ruby logger
696
- log = Logger.new(STDOUT)
697
- log.level = Logger::DEBUG
698
-
699
- SemanticLogger.default_level = :debug
700
- SemanticLogger.add_appender(log)
701
-
702
- logger = SemanticLogger['Example']
703
- logger.info "Hello World"
704
- logger.debug("Login time", :user => 'Joe', :duration => 100, :ip_address=>'127.0.0.1')
705
- ```
706
-
707
- It is recommended that every class or module have it's own logging instance.
708
- This can be achieved by including SemanticLogger::Loggable:
709
- ```ruby
710
- require 'semantic_logger'
711
- SemanticLogger.default_level = :trace
712
- SemanticLogger.add_appender('development.log')
713
-
714
- class ExternalSupplier
715
- # Makes available a class and instance level logger
716
- # ExternalSupplier.logger and ExternalSupplier#logger
717
- include SemanticLogger::Loggable
718
-
719
- # logger class method example
720
- def self.hostname=(hostname)
721
- logger.debug "Setting hostname to #{hostname}"
722
- @@hostname = hostname
723
- end
724
-
725
- # logger instance method example
726
- def call_supplier(amount, name)
727
- logger.debug "Calculating with amount", { :amount => amount, :name => name }
728
-
729
- # Measure and log on completion how long the call took to the external supplier
730
- logger.benchmark_info "Calling external interface" do
731
- # Code to call the external supplier ...
732
- end
733
- end
734
- end
735
- ```
736
-
737
- ### Logging to Syslog
738
-
739
- Log to a local Syslog:
740
- ```ruby
741
- require 'semantic_logger'
742
- SemanticLogger.default_level = :trace
743
- SemanticLogger.add_appender(SemanticLogger::Appender::Syslog.new)
744
- ```
745
-
746
- Log to a local file and to a remote Syslog server such as syslog-ng over TCP:
747
- ```ruby
748
- require 'semantic_logger'
749
- SemanticLogger.default_level = :trace
750
- SemanticLogger.add_appender('development.log')
751
- SemanticLogger.add_appender(SemanticLogger::Appender::Syslog.new(:server => 'tcp://myloghost:514'))
752
- ```
753
-
754
- ## Configuration
755
-
756
- The Semantic Logger follows the principle where multiple appenders can be active
757
- at the same time. For example, this allows one to log to MongoDB and the Rails
758
- log file at the same time.
759
-
760
- ### Rails Configuration
761
-
762
- To automatically replace the Rails logger with Semantic Logger use the gem [rails_semantic_logger](http://github.com/reidmorrison/rails_semantic_logger)
763
-
764
- ## Log Struct
765
-
766
- Internally all log messages are passed around in a Log Struct. In order
767
- to write your own custom formatter or log appender it is necessary to understand
768
- the fields:
769
-
770
- ```ruby
771
- Log = Struct.new(:level, :thread_name, :name, :message, :payload, :time, :duration, :tags, :level_index, :exception, :metric)
772
- ```
773
- level [Symbol]
774
-
775
- * Log level of the supplied log call
776
- * :trace, :debug, :info, :warn, :error, :fatal
777
-
778
- thread_name [String]
779
-
780
- * Name or id of the thread in which the logging call was called
781
-
782
- name [String]
783
-
784
- * Class name supplied to the logging instance
785
-
786
- message [String]
787
-
788
- * Text message to be logged
789
-
790
- payload [Hash|Exception]
791
-
792
- * Optional Hash or Ruby Exception object to be logged
793
-
794
- time [Time]
795
-
796
- * The time at which the log entry was created
797
-
798
- duration [Float]
799
-
800
- * The time taken in milli-seconds to complete a benchmark call
801
-
802
- tags [Array<String>]
803
-
804
- * Any tags active on the thread when the log call was made
805
-
806
- level_index [Integer]
807
-
808
- * Internal use only. Index of the log level
809
-
810
- metric [Object]
811
-
812
- * Object supplied when the benchmark api was called
813
-
814
- ### Mixing Logging Levels
815
-
816
- It is sometimes useful to log a subset of the log messages to a separate file
817
- or appender. For example, log :error and :fatal level messages to a special
818
- error file.
819
-
820
- Below is a stand-alone example that better shows this behavior:
821
-
822
- ```ruby
823
- require 'semantic_logger'
824
-
825
- # Set default log level for new logger instances
826
- SemanticLogger.default_level = :info
827
-
828
- # Log all warning messages and above to warnings.log
829
- SemanticLogger.add_appender('log/warnings.log', :warn)
830
-
831
- # Log all trace messages and above to trace.log
832
- SemanticLogger.add_appender('log/trace.log', :trace)
833
-
834
- logger = SemanticLogger['MyClass']
835
- logger.level = :trace
836
- logger.trace "This is a trace message"
837
- logger.info "This is an info message"
838
- logger.warn "This is a warning message"
839
- ```
840
-
841
- The output is as follows:
842
- ```bash
843
- ==> trace.log <==
844
- 2013-08-02 14:15:56.733532 T [35669:70176909690580] MyClass -- This is a trace message
845
- 2013-08-02 14:15:56.734273 I [35669:70176909690580] MyClass -- This is an info message
846
- 2013-08-02 14:15:56.735273 W [35669:70176909690580] MyClass -- This is a warning message
847
-
848
- ==> warnings.log <==
849
- 2013-08-02 14:15:56.735273 W [35669:70176909690580] MyClass -- This is a warning message
850
- ```
851
-
852
- ### Custom Formatters
853
-
854
- The formatting for each appender can be replaced with custom code. To replace the
855
- existing formatter supply a block of code when creating the appender.
856
-
857
- Example: Formatter that just returns the Log Struct
858
-
859
- ```ruby
860
- require 'semantic_logger'
861
-
862
- SemanticLogger.default_level = :trace
863
-
864
- SemanticLogger.add_appender(STDOUT) do |log|
865
- # This formatter just returns the log struct as a string
866
- log.inspect
867
- end
868
-
869
- logger = SemanticLogger['Hello']
870
- logger.info "Hello World"
871
- ```
872
- Output:
873
-
874
- #<struct SemanticLogger::Base::Log level=:info, thread_name=70167090649820, name="Hello", message="Hello World", payload=nil, time=2012-10-24 10:09:33 -0400, duration=nil, tags=nil, level_index=2>
875
-
876
-
877
- Example: Replace the default log file formatter
878
-
879
- ```ruby
880
- require 'semantic_logger'
881
- SemanticLogger.default_level = :trace
882
-
883
- SemanticLogger.add_appender('development.log') do |log|
884
- tags = log.tags.collect { |tag| "[#{tag}]" }.join(" ") + " " if log.tags && (log.tags.size > 0)
885
-
886
- message = log.message.to_s
887
- message << " -- " << log.payload.inspect if log.payload
888
- message << " -- " << "#{log.exception.class}: #{log.exception.message}\n#{(log.exception.backtrace || []).join("\n")}" if log.exception
889
-
890
- duration_str = log.duration ? "(#{'%.1f' % log.duration}ms) " : ''
891
-
892
- "#{SemanticLogger::Appender::Base.formatted_time(log.time)} #{log.level.to_s[0..0].upcase} [#{$$}:#{log.thread_name}] #{tags}#{duration_str}#{log.name} -- #{message}"
893
- end
894
- ```
895
-
896
- Example: Replace the default MongoDB formatter
897
-
898
- ```ruby
899
- # Log to MongoDB and supply a custom document formatter
900
- mongodb_appender = SemanticLogger::Appender::MongoDB.new(
901
- :db => Cache::Work.db,
902
- :collection_size => 25.gigabytes
903
- ) do |log|
904
- # Return a document (Hash) of the data to be saved to MongoDB
905
- document = {
906
- :time => log.time,
907
- :host_name => SemanticLogger::Appender::MongoDB.host_name,
908
- :pid => $PID,
909
- :thread_name => log.thread_name,
910
- :name => log.name,
911
- :level => log.level,
912
- :level_index => log.level_index,
913
- }
914
- document[:application] = 'MyApplication'
915
- document[:message] = SemanticLogger::Appender::MongoDB.strip_colorizing(log.message) if log.message
916
- document[:duration] = log.duration if log.duration
917
- document[:tags] = log.tags if log.tags && (log.tags.size > 0)
918
- document[:payload] = log.payload if log.payload
919
- document[:exception] = {
920
- :name => log.exception.class.name,
921
- :message => log.exception.message,
922
- :stack_trace => log.exception.backtrace
923
- } if log.exception
924
- document
925
- end
926
- SemanticLogger.add_appender(mongodb_appender)
927
- ```
928
-
929
- ## Performance
930
-
931
- The traditional logging implementations write their log information to file in the
932
- same thread of execution as the program itself. This means that for every log entry
933
- the program has to wait for the data to be written.
934
-
935
- With Semantic Logger it uses a dedicated thread for logging so that writing to
936
- the log file or other appenders does not hold up program execution.
937
-
938
- Also, since the logging is in this separate thread there is no impact to program
939
- execution if we decided to add another appender.
940
- For example, log to both a file and a MongoDB collection.
941
-
942
- ## Log Rotation
943
-
944
- Since the log file is not re-opened with every call, when the log file needs
945
- to be rotated, use a copy-truncate operation over deleting the file.
946
-
947
- ## Why Semantic logging?
948
-
949
- Just as there is the initiative to add Semantic information to data on the web
950
- so that computers can directly understand the content without having to resort
951
- to complex regular expressions or machine learning techniques, it is important
952
- to be able to do the same with log files or data.
953
-
954
- Semantic Logger allows every log entry to have not only a message, but a payload
955
- that can be written to a file or a NOSQL destination.
956
-
957
- Once the logging data is in the NOSQL data store it can be queried quickly and
958
- efficiently. Some SQL data stores also allow complex data types that could be used
959
- for storing and querying the logging data
960
-
961
- Before writing SemanticLogger all of the following logging frameworks were thoroughly
962
- evaluated. None of them met the above Semantic requirements, or the performance requirements
963
- of hundreds of threads all logging at the same time:
964
- logback, logging, log4r, central_logger, whoops
965
-
966
- ## Architecture & Performance
967
-
968
- In order to ensure that logging does not hinder the performance of the application
969
- all log entries are written to thread-safe Queue. A separate thread is responsible
970
- for writing the log entries to each of the appenders.
971
-
972
- In this way formatting and disk or network write delays will not affect the
973
- performance of the application. Also adding more than one appender does not affect
974
- the runtime performance of the application.
975
-
976
- The logging thread is automatically started on initialization. When the program
977
- terminates it will call flush on each of the appenders.
978
-
979
- Calling SemanticLogger::Logger#flush will wait until all outstanding log messages
980
- have been written and flushed to their respective appenders before returning.
981
-
982
- ## Write your own Appender
983
-
984
- To write your own appender it should meet the following requirements:
985
-
986
- * Inherit from SemanticLogger::Base
987
- * In the initializer connect to the resource being logged to
988
- * Implement #log(log) which needs to write to the relevant resource
989
- * Implement #flush if the resource can be flushed
990
- * Write a test for the new appender
991
-
992
- The #log method takes the log struct as a parameter which is described above.
993
-
994
- Basic outline for an Appender:
995
-
996
- ```ruby
997
- require 'semantic_logger'
998
-
999
- class SimpleAppender < SemanticLogger::Appender::Base
1000
- def initialize(level=nil, &block)
1001
- # Set the log level and formatter if supplied
1002
- super(level, &block)
1003
- end
1004
-
1005
- # Display the log struct and the text formatted output
1006
- def log(log)
1007
- p log
1008
- puts formatter.call(log)
1009
- end
1010
-
1011
- # Optional
1012
- def flush
1013
- puts "Flush :)"
1014
- end
1015
- end
1016
- ```
1017
-
1018
- Sample program calling the above appender:
1019
- ```ruby
1020
- SemanticLogger.default_level = :trace
1021
- # Log to file dev.log
1022
- SemanticLogger.add_appender('dev.log')
1023
- # Also log the above sample appender
1024
- SemanticLogger.add_appender(SimpleAppender.new)
1025
-
1026
- logger = SemanticLogger['Hello']
1027
- logger.info "Hello World"
1028
- ```
1029
-
1030
- Look at the [existing appenders](https://github.com/reidmorrison/semantic_logger/tree/master/lib/semantic_logger/appender) for good examples
1031
-
1032
- To have your appender included in the standard list of appenders, submit it along
1033
- with complete working tests.
1034
- See the [MongoDB Appender Test](https://github.com/reidmorrison/semantic_logger/blob/master/test/appender_mongodb_test.rb) for an example.
10
+ For complete documentation see: http://reidmorrison.github.io/semantic_logger
1035
11
 
1036
12
  ## Dependencies
1037
13
 
@@ -1053,8 +29,7 @@ As of SemanticLogger V2.0 the Rails logging is no longer automatically replaced
1053
29
  when including SemanticLogger. Include the [rails_semantic_logger](http://github.com/reidmorrison/rails_semantic_logger)
1054
30
  gem to replace the Rails default logger with SemanticLogger
1055
31
 
1056
- Meta
1057
- ----
32
+ ## Meta
1058
33
 
1059
34
  * Code: `git clone git://github.com/reidmorrison/semantic_logger.git`
1060
35
  * Home: <https://github.com/reidmorrison/semantic_logger>
@@ -1063,18 +38,15 @@ Meta
1063
38
 
1064
39
  This project uses [Semantic Versioning](http://semver.org/).
1065
40
 
1066
- Author
1067
- -------
41
+ ## Author
1068
42
 
1069
43
  Reid Morrison :: reidmo@gmail.com :: @reidmorrison
1070
44
 
1071
- Contributors
1072
- ------------
45
+ ## Contributors
1073
46
 
1074
47
  Marc Bellingrath :: marrrc.b@gmail.com
1075
48
 
1076
- License
1077
- -------
49
+ ## License
1078
50
 
1079
51
  Copyright 2012, 2013, 2014 Reid Morrison
1080
52