semantic_logger 2.8.0 → 2.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 51a1e44ec07ae96cd5dad5b3a3dc2c6377960b15
4
- data.tar.gz: 820304cb8615f71b1f921869fcc70e0766ef7258
3
+ metadata.gz: 05fe9c7ccd2177383e5f4a8cf4b5b6d0bf84df71
4
+ data.tar.gz: 0516f2bb270d5ea5d78b67217fd02745ebe4c3ce
5
5
  SHA512:
6
- metadata.gz: 3faa5b434bc505b9d2045f7f00c91feae01f89bcd3a498ea7e0fe8f831d75fd276f05fd059ddd149ba2755d3ab6dfa3abeddc6d9c025191e75a4517c304c155f
7
- data.tar.gz: d0691ea0ad358cf91c266e5e9005d510aed493f4adb8255423a425133c7258248d53482420e06623db1009a620adcd50ec616c56ad687757a4427eabbc16c603
6
+ metadata.gz: 39f2f6909cdd7e1623fd099226849a2086873b52334ddabbde84067a12b3e88d1058384b346f09c11f1c3c357b38758ae6e51dd649e3d39f2765304f40ac59d5
7
+ data.tar.gz: 025cca0bde8b1733a023ba9139b474d9f14e85444b518e24764b2a346aa8676be5441e4e006866caf8a103027156e4c39f8b30af3cfd144d93a2fc96f26840ed
data/LICENSE.txt CHANGED
@@ -186,7 +186,7 @@
186
186
  same "printed page" as the copyright notice for easier
187
187
  identification within third-party archives.
188
188
 
189
- Copyright 2012, 2013 Reid Morrison
189
+ Copyright 2012, 2013, 2014 Reid Morrison
190
190
 
191
191
  Licensed under the Apache License, Version 2.0 (the "License");
192
192
  you may not use this file except in compliance with the License.
data/README.md CHANGED
@@ -1,1037 +1,13 @@
1
1
  semantic_logger [![Build Status](https://secure.travis-ci.org/reidmorrison/semantic_logger.png?branch=master)](http://travis-ci.org/reidmorrison/semantic_logger)
2
2
  ===============
3
3
 
4
- Scalable, next generation logging for Ruby
4
+ Next generation logging system for Ruby to support highly concurrent, high throughput, low latency systems
5
5
 
6
6
  * http://github.com/reidmorrison/semantic_logger
7
7
 
8
- ## Overview
8
+ ## Documentation
9
9
 
10
- Semantic Logger takes logging in Ruby to the next level by adding several new
11
- capabilities to the commonly used Logging API:
12
-
13
- High Performance
14
-
15
- * Logging is performed in a separate thread so as not to slow down the application
16
- whilst logging to one or more destinations
17
- * Supports logging from hundreds of concurrent threads in the same process
18
-
19
- Drop-in Replacement
20
-
21
- * Simple drop-in replacement for the Ruby, or the Rails loggers
22
- * Supports current common logging interface
23
- * No changes to existing to code to use new logger ( other than replacing the logger )
24
-
25
- Dynamic
26
-
27
- * Increase the log level at runtime for just one class
28
- * For example enable debug level logging for a single class (logging instance)
29
- while the program is running to get more detailed logging in production for just that class
30
- * Change the default global logging level for all classes, unless that class has
31
- specifically been overridden above
32
- * Use UNIX signals to change the log level for a running process
33
-
34
- Tagged Logging
35
-
36
- * Supply custom data to be added to every log entry within a block of code,
37
- including libraries and existing gems
38
- * Tagged logging is critical for any high traffic site so that one can narrow
39
- down log entries for a single call that is mixed in with log entries
40
- from hundreds of other log entries
41
-
42
- Payload support
43
-
44
- * Aside from the regular log message, a hash payload can also be supplied with
45
- every log entry
46
- * Very powerful when logging to NOSQL destinations that allow queries against
47
- any data in the payload
48
-
49
- Exceptions
50
-
51
- * Directly log exceptions
52
- * Semantic Logger standardizes the logging of exceptions with their backtraces
53
- to text destinations and writes the exception elements as a hash to NOSQL
54
- destinations
55
-
56
- Benchmarking
57
-
58
- * The performance of any block of code can be measured and logged at the same time
59
- depending on the active log level
60
- * Supports only logging when the block of code exceeds a specified number of milli-seconds.
61
- Makes it easy to find bottlenecks when the system suddenly slows down in production
62
- * Exceptions thrown in the block of code can also be logged so as to aid in finding
63
- exceptions that may be discarded or hidden by the application
64
- * Benchmarked data can also be forwarded to external metric systems so that the
65
- performance of these blocks can be measured and/or monitored over time
66
-
67
- Thread Safe
68
-
69
- * Semantic Logger is completely thread safe and all methods can be called
70
- concurrently from any thread
71
- * Tagged logging keeps any tagging data on a per-thread basis to ensure that
72
- tags from different threads are not inter-mingled
73
- * Supports highly concurrent environments running hundreds of threads
74
- * Each appender writes all log entries sequentially in the appender thread so
75
- that log entries are written in the correct sequence
76
- * Avoids issues that other loggers experience when multiple threads try to write
77
- to the same log file at the same time creating partial and overwritten log
78
- entries in the log file
79
-
80
- Thread Aware
81
-
82
- * Includes the process id, and thread name or thread id in every log entry so that
83
- log entries from different processes and even threads are easily discernable
84
- * Human readable names can be assigned to every thread for logging purposes
85
-
86
- Trace Level
87
-
88
- * :trace is a new level common in other languages and is commonly used for
89
- logging trace level detail. It is intended for logging data at level below
90
- :debug.
91
- * :trace can be used for logging the actual data sent or received over the network
92
- that is rarely needed but is critical when things are not working as expected.
93
- * Since :trace can be enabled on a per class basis it can even be turned on
94
- in production to resolve what was actually sent to an external vendor
95
-
96
- Multiple Destinations
97
-
98
- * Log to multiple destinations at the same time ( File and MongoDB, etc.. )
99
- * Each destination can also have its own log level.
100
- For example, only log :info and above to MongoDB, or :warn and above to a
101
- second log file
102
-
103
- Semantic Capabilities
104
-
105
- * With Semantic Logger it is simple to mix-in additional semantic information with
106
- every log entry
107
- * The application or class name is automatically included for every log entry under
108
- a specific logging instance
109
- * Includes the duration of blocks of code
110
- * Any hash containing context specific information such as user_id or location information
111
-
112
- Beyond Tagged Logging
113
-
114
- * Supply entire hash of custom data to be added to the payload of every log entry
115
- within a block of code, including libraries and existing gems
116
-
117
- NOSQL Destinations
118
-
119
- * Every log entry is broken down into elements that NOSQL data stores can understand:
120
-
121
- ```json
122
- {
123
- "_id" : ObjectId("5034fa48e3f3fea945e83ef2"),
124
- "time" : ISODate("2012-08-22T15:27:04.409Z"),
125
- "host_name" : "release",
126
- "pid" : 16112,
127
- "thread_name" : "main",
128
- "name" : "UserLocator",
129
- "level" : "debug",
130
- "message" : "Fetch user information",
131
- "duration" : 12,
132
- "payload" : {
133
- "user" : "Jack",
134
- "zip_code" : 12345,
135
- "location" : "US"
136
- }
137
- }
138
- ```
139
-
140
- Customizable
141
-
142
- * Custom formatting by destination
143
- * Easy to "roll your own" destination (Appender).
144
- For example to log to Hadoop, Redis, etc..
145
-
146
- ## Introduction
147
-
148
- Semantic Logger is a Logger that supports logging of meta-data, along with text messages
149
- to multiple appenders
150
-
151
- An appender is a Logging destination such as a File, MongoDB collection, etc..
152
- Multiple Appenders can be active at the same time. All log entries are written
153
- to each appender.
154
-
155
- Machines can understand the logged data without having to use
156
- complex Regular Expressions or other text parsing techniques
157
-
158
- Semantic Logger, sits on top of existing logger implementations and can also
159
- be used as a drop in replacement for existing Ruby loggers.
160
- This allows the existing logging to be replaced immediately with the
161
- Semantic Logger Appenders, and over time the calls can be replaced with ones
162
- that contain the necessary meta-data.
163
-
164
- Example of current calls:
165
-
166
- ```ruby
167
- logger.info("Queried users table in #{duration} ms, with a result code of #{result}")
168
- ```
169
-
170
- For a machine to find all queries for table 'users' that took longer than
171
- 100 ms, would require using a regular expression just to extract the table name
172
- and duration, then apply the necessary logic. It also assumes that the text
173
- is not changed and that matches will not be found when another log entry has
174
- similar text output.
175
-
176
- This can be changed over time to:
177
-
178
- ```ruby
179
- logger.info("Queried table",
180
- :duration => duration,
181
- :result => result,
182
- :table => "users",
183
- :action => "query")
184
- ```
185
-
186
- Using the MongoDB appender, we can easily find all queries for table 'users'
187
- that took longer than 100 ms:
188
-
189
- ```javascript
190
- db.logs.find({"payload.table":"users", "payload.action":"query", "payload.duration":{$gt:100} })
191
- ```
192
-
193
- Since Semantic Logger can call existing Loggers, it does not force end-users
194
- to have to adopt a Semantic aware adapter. Although, such adapters create
195
- tremendous value in the problem monitoring and determination processes.
196
-
197
- ## Logging API
198
-
199
- ### Standard Logging methods
200
-
201
- The Semantic Logger logging API supports the existing logging interface for
202
- the Rails and Ruby Loggers. For example:
203
-
204
- ```ruby
205
- logger.info("Hello World")
206
- ```
207
-
208
- Or to query whether a specific log level is set
209
-
210
- ```ruby
211
- logger.info?
212
- ```
213
-
214
- The following traditional logging methods are available
215
-
216
- ```ruby
217
- logger.trace("Low level trace information such as data sent over a socket")
218
- logger.debug("Debugging information to aid with problem determination")
219
- logger.info("Informational message such as request received")
220
- logger.warn("Warn about something in the system")
221
- logger.error("An error occurred during processing")
222
- logger.fatal("Oh no something really bad happened")
223
- ```
224
-
225
- Each of the above calls can take additional parameters, for example:
226
-
227
- ```ruby
228
- log.info(message, payload=nil, exception=nil, &block)
229
- ```
230
-
231
- Parameters
232
-
233
- - message: The text message to log.
234
- Mandatory only if no block is supplied
235
- - payload: Optional, either a Ruby Exception object or a Hash
236
- - exception: Optional, Ruby Exception object. Allows both an exception and a payload to be logged
237
- - block: The optional block is executed only if the corresponding log level
238
- is active. Can be used to prevent unnecessary calculations of debug data in
239
- production.
240
-
241
- Examples:
242
-
243
- ```ruby
244
- logger.debug("Calling Supplier")
245
-
246
- logger.debug("Calling Supplier", :request => 'update', :user => 'Jack')
247
-
248
- logger.debug { "A total of #{result.inject(0) {|sum, i| i+sum }} were processed" }
249
- ```
250
-
251
- ## Exceptions
252
-
253
- The Semantic Logger adds an optional parameter to the existing log methods so that
254
- a corresponding Exception can be logged in a standard way
255
-
256
- ```ruby
257
- begin
258
- # ... Code that can raise an exception
259
- rescue Exception => exception
260
- logger.error("Oops external call failed", exception)
261
- # Re-raise or handle the exception
262
- raise exception
263
- end
264
- ```
265
-
266
- ### Payload
267
-
268
- The Semantic Logger adds an extra parameter to the existing log methods so that
269
- additional payload can be logged, such as a Hash or a Ruby Exception object.
270
-
271
- ```ruby
272
- logger.info("Oops external call failed", :result => :failed, :reason_code => -10)
273
- ```
274
-
275
- The additional payload is machine readable so that we don't have to write complex
276
- regular expressions so that a program can analyze log output. With the MongoDB
277
- appender the payload is written directly to MongoDB as part of the document and
278
- is therefore fully searchable
279
-
280
- ### Benchmarking
281
-
282
- Another common logging requirement is to measure the time it takes to execute a block
283
- of code based on the log level. For example:
284
-
285
- ```ruby
286
- Rails.logger.benchmark_info "Calling external interface" do
287
- # Code to call external service ...
288
- end
289
- ```
290
-
291
- The following output will be written to file:
292
-
293
- 2012-08-30 15:37:29.474 I [48308:ScriptThreadProcess: script/rails] (5.2ms) Rails -- Calling external interface
294
-
295
- If an exception is raised during the block the exception is logged
296
- at the same log level as the benchmark along with the duration and message.
297
- The exception will flow through to the caller unchanged
298
-
299
- The following benchmarking methods are available
300
-
301
- ```ruby
302
- logger.benchmark_trace("Low level trace information such as data sent over a socket")
303
- logger.benchmark_debug("Debugging information to aid with problem determination")
304
- logger.benchmark_info("Informational message such as request received")
305
- logger.benchmark_warn("Warn about something in the system")
306
- logger.benchmark_error("An error occurred during processing")
307
- logger.benchmark_fatal("Oh no something really bad happened")
308
- logger.benchmark(:info, "Informational message such as request received")
309
- ```
310
-
311
- Each of the above calls can take additional parameters, for example:
312
-
313
- ```ruby
314
- log.benchmark_info(message, params=nil) do
315
- # Measure how long it takes to run this block of code
316
- end
317
- ```
318
-
319
- Parameters
320
-
321
- - message: The mandatory text message to log.
322
- - params:
323
- ```
324
- :log_exception
325
- Control whether or how an exception thrown in the block is
326
- reported by Semantic Logger. Values:
327
- :full
328
- Log the exception class, message, and backtrace
329
- :partial
330
- Log the exception class and message
331
- The backtrace will not be logged
332
- :off
333
- Any unhandled exception raised in the block will not be logged
334
- Default: :partial
335
-
336
- :min_duration [Float]
337
- Only log if the block takes longer than this duration in ms
338
- Default: 0.0
339
-
340
- :payload [Hash]
341
- Optional, Hash payload
342
-
343
- :exception [Exception]
344
- Optional, Ruby Exception object to log along with the duration of the supplied block
345
-
346
- :duration [Float]
347
- Optional, supply the duration in ms that is logged when a block is not supplied
348
- If a block is not supplied then :duration is mandatory
349
- If a block is supplied :duration is ignored
350
-
351
- :metric [Object]
352
- Optional, when this parameter is supplied all subscribers will be notified of this
353
- metric, along with the Log Struct described below
354
- ```
355
-
356
- ### Logging levels
357
-
358
- The following logging levels are available through Semantic Logger
359
-
360
- :trace, :debug, :info, :warn, :error, :fatal
361
-
362
- The log levels are listed above in the order of precedence with the most detail to the least.
363
- For example :debug would include :info, :warn, :error, :fatal levels but not :trace
364
- And :fatal would only log :fatal error messages and nothing else
365
-
366
- :unknown has been mapped to :fatal for Rails and Ruby Logger
367
-
368
- :trace is a new level that is often used for tracing low level calls such
369
- as the data sent or received to external web services. It is also commonly used
370
- in the development environment for low level trace logging of methods calls etc.
371
-
372
- If only the rails logger is being used, then :trace level calls will be logged
373
- as debug calls only if the log level is set to trace
374
-
375
- ### Changing the Class name for Log Entries
376
-
377
- When Semantic Logger is included in a Rails project it automatically replaces the
378
- loggers for Rails, ActiveRecord::Base, ActionController::Base, and ActiveResource::Base
379
- with wrappers that set their Class name. For example in semantic_logger/railtie.rb:
380
-
381
- ```ruby
382
- ActiveRecord::Base.logger = SemanticLogger[ActiveRecord]
383
- ```
384
-
385
- By replacing their loggers we now get the class name in the text logging output:
386
-
387
- 2012-08-30 15:24:13.439 D [47900:main] ActiveRecord -- SQL (12.0ms) SELECT `schema_migrations`.`version` FROM `schema_migrations`
388
-
389
- It is recommended to include a class specific logger for all major classes that will
390
- be logging using the SemanticLogger::Loggable mix-in. For Example:
391
-
392
- ```ruby
393
- class ExternalSupplier
394
- # Lazy load logger class variable on first use
395
- include SemanticLogger::Loggable
396
-
397
- def call_supplier(amount, name)
398
- logger.debug "Calculating with amount", { :amount => amount, :name => name }
399
-
400
- # Measure and log on completion how long the call took to the external supplier
401
- logger.benchmark_info "Calling external interface" do
402
- # Code to call the external supplier ...
403
- end
404
- end
405
- end
406
- ```
407
-
408
- This will result in the log output identifying the log entry as from the ExternalSupplier class
409
-
410
- 2012-08-30 15:37:29.474 I [48308:ScriptThreadProcess: script/rails] (5.2ms) ExternalSupplier -- Calling external interface
411
-
412
- ### Changing the log level for a single class at runtime
413
-
414
- Since the logger is class specific, its log level can be changed dynamically at runtime.
415
- For example, to temporarily set the log level to :trace to diagnose an issue:
416
-
417
- ```ruby
418
- require 'semantic_logger'
419
-
420
- SemanticLogger.default_level = :info
421
- SemanticLogger.add_appender('example.log')
422
-
423
- class ExternalSupplier
424
- # Lazy load logger class variable on first use
425
- include SemanticLogger::Loggable
426
-
427
- def call_supplier(amount, name)
428
- logger.trace "Calculating with amount", { :amount => amount, :name => name }
429
-
430
- # Measure and log on completion how long the call took to the external supplier
431
- logger.benchmark_info "Calling external interface" do
432
- # Code to call the external supplier ...
433
- end
434
- end
435
- end
436
-
437
- # Create and use the class
438
- supplier = ExternalSupplier.new
439
- supplier.call_supplier(100, 'Jack')
440
-
441
- # Now change the log level to :trace
442
- ExternalSupplier.logger.level = :trace
443
-
444
- # Call the supplier, this time including trace level messages
445
- supplier.call_supplier(100, 'Jack')
446
-
447
- # Change the log level back to the default level
448
- ExternalSupplier.logger.level = SemanticLogger.default_level
449
- ```
450
-
451
- Below is the output from the above example showing the :trace log level message
452
- that was written during the second call to the ExternalSupplier:
453
-
454
- ```
455
- 2013-11-07 16:19:26.496 I [35674:main] (0.0ms) ExternalSupplier -- Calling external interface
456
-
457
- 2013-11-07 16:19:26.683 T [35674:main] ExternalSupplier -- Calculating with amount -- {:amount=>100, :name=>"Jack"}
458
- 2013-11-07 16:19:26.683 I [35674:main] (0.0ms) ExternalSupplier -- Calling external interface
459
- ```
460
-
461
- ### Change the global default logging level at runtime
462
-
463
- Log levels can be changed using signals on operating systems that support them.
464
- This allows log levels to be changed externally without requiring a restart
465
- of the running process.
466
-
467
- When the signal is raised, the global default log level rotates through the following
468
- log levels in the following order, starting from the current global default level:
469
-
470
- ```ruby
471
- :warn, :info, :debug, :trace
472
- ```
473
-
474
- If the current level is :trace it wraps around back to :warn
475
-
476
- Example:
477
-
478
- ```
479
- kill -SIGUSR2 1234
480
- ```
481
-
482
- #### Enabling Log Level Signal handler
483
-
484
- On startup SemanticLogger does not register any signals so that it does not
485
- interfere with any existing signal handlers. In order to enable the above log level
486
- changes the signal handler must be registered by calling `SemanticLogger.add_signal_handler`
487
-
488
- ```ruby
489
- require 'semantic_logger'
490
-
491
- # Enable signal handling for this process
492
- SemanticLogger.add_signal_handler('USR2')
493
-
494
- SemanticLogger.add_appender('development.log')
495
-
496
- logger = SemanticLogger['Example']
497
- logger.info "Hello World"
498
- ```
499
-
500
- Note: The changes to the logging level will not change for any classes where the
501
- log_level was set explicity within the application itself. The above signal only changes
502
- the global default level, which is used by loggers when their log level has not been changed.
503
-
504
- #### Change the log level without using signals
505
-
506
- If the application has another means of communicating without needing signals,
507
- the global default log level can be modified using `SemanticLogger.default_level=`
508
-
509
- ```ruby
510
- # Change the global default logging level for active loggers
511
- SemanticLogger.default_level = :debug
512
- ```
513
-
514
- ### Tagged Logging
515
-
516
- Semantic Logger allows any Ruby or Rails program to also include tagged logging.
517
-
518
- This means that any logging performed within a block, including any called
519
- libraries or gems to include the specified tag with every log entry.
520
-
521
- Using Tagged logging is critical in any highly concurrent environment so that
522
- one can quickly find all related log entries across all levels of code, and even
523
- across threads
524
-
525
- ```ruby
526
- logger.tagged(tracking_number) do
527
- logger.debug("Hello World")
528
- # ...
529
- end
530
- ```
531
-
532
- ### Beyond Tagged Logging
533
-
534
- Blocks of code can be tagged with not only values, but can be tagged with
535
- entire hashes of data. The additional hash of data will be merged into
536
- the payload of every log entry
537
-
538
- For example every corresponding log entry could include a hash containing
539
- a user_id, name, region, zip_code, tracking_number, etc...
540
-
541
- ```ruby
542
- logger.with_payload(:user => 'Jack', :zip_code => 12345) do
543
- logger.debug("Hello World")
544
- # ...
545
- end
546
- ```
547
-
548
- ### Named threads
549
-
550
- SemanticLogger logs the name or id of the thread in every log message.
551
-
552
- On Ruby MRI the thread name is by default the thread's object_id, For example: 70184354571980
553
-
554
- ```
555
- 2013-11-07 16:25:14.279627 I [35841:70184354571980] (0.0ms) ExternalSupplier -- Calling external interface
556
- ```
557
-
558
- To set a custom name for any thread so that it shows up in the logger:
559
-
560
- ```ruby
561
- Thread.current.name = "User calculation thread 32"
562
- ```
563
-
564
- Sample output:
565
-
566
- ```
567
- 2013-11-07 16:26:02.744139 I [35841:User calculation thread 32] (0.0ms) ExternalSupplier -- Calling external interface
568
- ```
569
-
570
- When running JRuby, Thread.current.name will also set the underlying thread name in the JVM
571
- which is very useful when monitoring the JVM via JMX using tools such as jconsole.
572
-
573
- #### NOTE:
574
-
575
- Make sure that the assigned thread name is unique otherwise it will be difficult
576
- to distinguish between concurrently running threads if they have the same name.
577
-
578
- For example, use the current thread object_id to ensure uniqueness:
579
-
580
- ```ruby
581
- Thread.current.name = "Worker Thread:#{Thread.current.object_id}"
582
- ```
583
-
584
- ### Metrics integration
585
-
586
- In production environments it is often necessary to not only measure the performance of a
587
- block of code using for example:
588
-
589
- ```ruby
590
- logger.benchmark_info "Calling external interface" do
591
- # Code to call the external supplier ...
592
- end
593
- ```
594
-
595
- A single subscriber can be defined to collect all the metrics and forward them
596
- for example to NewRelic:
597
-
598
- ```ruby
599
- # config/initializers/semantic_logger_metrics.rb
600
- SemanticLogger.on_metric do |log_struct|
601
- ::NewRelic::Agent.record_metric(log_struct.metric, log_struct.duration)
602
- end
603
- ```
604
-
605
- Add the :metric option to the log entry as follows:
606
-
607
- ```ruby
608
- logger.benchmark_info "Calling external interface", :metric => 'Custom/slow_action/beginning_work' do
609
- # Code to call the external supplier ...
610
- end
611
- ```
612
-
613
- ## Standalone SemanticLogger
614
-
615
- When using SemanticLogger inside of Rails all we need to do is include the
616
- rails_semantic_logger gem and the default Rails logger will be replaced with
617
- Semantic Logger.
618
-
619
- In a stand-alone or non-rails environment we can easily log to a file called
620
- 'development.log' as follows:
621
- ```ruby
622
- require 'semantic_logger'
623
- SemanticLogger.add_appender('development.log')
624
-
625
- logger = SemanticLogger['Example']
626
- logger.info "Hello World"
627
- ```
628
-
629
- By default it will only log :info and above, to log everything to the log file:
630
- ```ruby
631
- require 'semantic_logger'
632
- SemanticLogger.default_level = :trace
633
- SemanticLogger.add_appender('development.log')
634
-
635
- logger = SemanticLogger['Example']
636
- logger.info "Hello World"
637
- logger.trace "Low level trace information"
638
- ```
639
-
640
- By supplying multiple appenders Semantic Logger can write to multiple destinations
641
- at the same time. For example, log to a file and the screen:
642
- ```ruby
643
- require 'semantic_logger'
644
- SemanticLogger.default_level = :trace
645
- SemanticLogger.add_appender('development.log')
646
- SemanticLogger.add_appender(STDOUT)
647
-
648
- logger = SemanticLogger['Example']
649
- logger.info "Hello World"
650
- logger.trace "Low level trace information"
651
- ```
652
-
653
- To reduce the log level of logging to STDOUT to just :info and above, add the
654
- log_level such as :info as the second parameter when adding the appender:
655
- ```ruby
656
- require 'semantic_logger'
657
- SemanticLogger.default_level = :trace
658
- SemanticLogger.add_appender('development.log')
659
- SemanticLogger.add_appender(STDOUT, :info)
660
-
661
- logger = SemanticLogger['Example']
662
- logger.info "Hello World"
663
- logger.trace "Low level trace information"
664
- ```
665
-
666
- To log :debug and above to a log file, :error and above to $stderr, and :info
667
- and above to MongoDB:
668
- ```ruby
669
- require 'semantic_logger'
670
- require 'mongo'
671
-
672
- SemanticLogger.default_level = :debug
673
- SemanticLogger.add_appender('development.log')
674
- SemanticLogger.add_appender($stderr, :error)
675
-
676
- mongo_appender = SemanticLogger::Appender::MongoDB.new(
677
- :db => Mongodb::Connection.new['production_logging'],
678
- :collection_size => 25.gigabytes
679
- )
680
- SemanticLogger.add_appender(mongo_appender, :info)
681
-
682
- logger = SemanticLogger['Example']
683
- logger.info "Hello World"
684
- logger.trace "Low level trace information"
685
- logger.error "Oops an error occurred"
686
- logger.info("Login time", :user => 'Mary', :duration => 230, :ip_address=>'192.168.0.1')
687
- ```
688
-
689
- When starting out with Semantic Logger it can be useful to gain all the benefits
690
- of Semantic Logger and still continue to log to an existing logger:
691
- ```ruby
692
- require 'logger'
693
- require 'semantic_logger'
694
-
695
- # Built-in Ruby logger
696
- log = Logger.new(STDOUT)
697
- log.level = Logger::DEBUG
698
-
699
- SemanticLogger.default_level = :debug
700
- SemanticLogger.add_appender(log)
701
-
702
- logger = SemanticLogger['Example']
703
- logger.info "Hello World"
704
- logger.debug("Login time", :user => 'Joe', :duration => 100, :ip_address=>'127.0.0.1')
705
- ```
706
-
707
- It is recommended that every class or module have it's own logging instance.
708
- This can be achieved by including SemanticLogger::Loggable:
709
- ```ruby
710
- require 'semantic_logger'
711
- SemanticLogger.default_level = :trace
712
- SemanticLogger.add_appender('development.log')
713
-
714
- class ExternalSupplier
715
- # Makes available a class and instance level logger
716
- # ExternalSupplier.logger and ExternalSupplier#logger
717
- include SemanticLogger::Loggable
718
-
719
- # logger class method example
720
- def self.hostname=(hostname)
721
- logger.debug "Setting hostname to #{hostname}"
722
- @@hostname = hostname
723
- end
724
-
725
- # logger instance method example
726
- def call_supplier(amount, name)
727
- logger.debug "Calculating with amount", { :amount => amount, :name => name }
728
-
729
- # Measure and log on completion how long the call took to the external supplier
730
- logger.benchmark_info "Calling external interface" do
731
- # Code to call the external supplier ...
732
- end
733
- end
734
- end
735
- ```
736
-
737
- ### Logging to Syslog
738
-
739
- Log to a local Syslog:
740
- ```ruby
741
- require 'semantic_logger'
742
- SemanticLogger.default_level = :trace
743
- SemanticLogger.add_appender(SemanticLogger::Appender::Syslog.new)
744
- ```
745
-
746
- Log to a local file and to a remote Syslog server such as syslog-ng over TCP:
747
- ```ruby
748
- require 'semantic_logger'
749
- SemanticLogger.default_level = :trace
750
- SemanticLogger.add_appender('development.log')
751
- SemanticLogger.add_appender(SemanticLogger::Appender::Syslog.new(:server => 'tcp://myloghost:514'))
752
- ```
753
-
754
- ## Configuration
755
-
756
- The Semantic Logger follows the principle where multiple appenders can be active
757
- at the same time. For example, this allows one to log to MongoDB and the Rails
758
- log file at the same time.
759
-
760
- ### Rails Configuration
761
-
762
- To automatically replace the Rails logger with Semantic Logger use the gem [rails_semantic_logger](http://github.com/reidmorrison/rails_semantic_logger)
763
-
764
- ## Log Struct
765
-
766
- Internally all log messages are passed around in a Log Struct. In order
767
- to write your own custom formatter or log appender it is necessary to understand
768
- the fields:
769
-
770
- ```ruby
771
- Log = Struct.new(:level, :thread_name, :name, :message, :payload, :time, :duration, :tags, :level_index, :exception, :metric)
772
- ```
773
- level [Symbol]
774
-
775
- * Log level of the supplied log call
776
- * :trace, :debug, :info, :warn, :error, :fatal
777
-
778
- thread_name [String]
779
-
780
- * Name or id of the thread in which the logging call was called
781
-
782
- name [String]
783
-
784
- * Class name supplied to the logging instance
785
-
786
- message [String]
787
-
788
- * Text message to be logged
789
-
790
- payload [Hash|Exception]
791
-
792
- * Optional Hash or Ruby Exception object to be logged
793
-
794
- time [Time]
795
-
796
- * The time at which the log entry was created
797
-
798
- duration [Float]
799
-
800
- * The time taken in milli-seconds to complete a benchmark call
801
-
802
- tags [Array<String>]
803
-
804
- * Any tags active on the thread when the log call was made
805
-
806
- level_index [Integer]
807
-
808
- * Internal use only. Index of the log level
809
-
810
- metric [Object]
811
-
812
- * Object supplied when the benchmark api was called
813
-
814
- ### Mixing Logging Levels
815
-
816
- It is sometimes useful to log a subset of the log messages to a separate file
817
- or appender. For example, log :error and :fatal level messages to a special
818
- error file.
819
-
820
- Below is a stand-alone example that better shows this behavior:
821
-
822
- ```ruby
823
- require 'semantic_logger'
824
-
825
- # Set default log level for new logger instances
826
- SemanticLogger.default_level = :info
827
-
828
- # Log all warning messages and above to warnings.log
829
- SemanticLogger.add_appender('log/warnings.log', :warn)
830
-
831
- # Log all trace messages and above to trace.log
832
- SemanticLogger.add_appender('log/trace.log', :trace)
833
-
834
- logger = SemanticLogger['MyClass']
835
- logger.level = :trace
836
- logger.trace "This is a trace message"
837
- logger.info "This is an info message"
838
- logger.warn "This is a warning message"
839
- ```
840
-
841
- The output is as follows:
842
- ```bash
843
- ==> trace.log <==
844
- 2013-08-02 14:15:56.733532 T [35669:70176909690580] MyClass -- This is a trace message
845
- 2013-08-02 14:15:56.734273 I [35669:70176909690580] MyClass -- This is an info message
846
- 2013-08-02 14:15:56.735273 W [35669:70176909690580] MyClass -- This is a warning message
847
-
848
- ==> warnings.log <==
849
- 2013-08-02 14:15:56.735273 W [35669:70176909690580] MyClass -- This is a warning message
850
- ```
851
-
852
- ### Custom Formatters
853
-
854
- The formatting for each appender can be replaced with custom code. To replace the
855
- existing formatter supply a block of code when creating the appender.
856
-
857
- Example: Formatter that just returns the Log Struct
858
-
859
- ```ruby
860
- require 'semantic_logger'
861
-
862
- SemanticLogger.default_level = :trace
863
-
864
- SemanticLogger.add_appender(STDOUT) do |log|
865
- # This formatter just returns the log struct as a string
866
- log.inspect
867
- end
868
-
869
- logger = SemanticLogger['Hello']
870
- logger.info "Hello World"
871
- ```
872
- Output:
873
-
874
- #<struct SemanticLogger::Base::Log level=:info, thread_name=70167090649820, name="Hello", message="Hello World", payload=nil, time=2012-10-24 10:09:33 -0400, duration=nil, tags=nil, level_index=2>
875
-
876
-
877
- Example: Replace the default log file formatter
878
-
879
- ```ruby
880
- require 'semantic_logger'
881
- SemanticLogger.default_level = :trace
882
-
883
- SemanticLogger.add_appender('development.log') do |log|
884
- tags = log.tags.collect { |tag| "[#{tag}]" }.join(" ") + " " if log.tags && (log.tags.size > 0)
885
-
886
- message = log.message.to_s
887
- message << " -- " << log.payload.inspect if log.payload
888
- message << " -- " << "#{log.exception.class}: #{log.exception.message}\n#{(log.exception.backtrace || []).join("\n")}" if log.exception
889
-
890
- duration_str = log.duration ? "(#{'%.1f' % log.duration}ms) " : ''
891
-
892
- "#{SemanticLogger::Appender::Base.formatted_time(log.time)} #{log.level.to_s[0..0].upcase} [#{$$}:#{log.thread_name}] #{tags}#{duration_str}#{log.name} -- #{message}"
893
- end
894
- ```
895
-
896
- Example: Replace the default MongoDB formatter
897
-
898
- ```ruby
899
- # Log to MongoDB and supply a custom document formatter
900
- mongodb_appender = SemanticLogger::Appender::MongoDB.new(
901
- :db => Cache::Work.db,
902
- :collection_size => 25.gigabytes
903
- ) do |log|
904
- # Return a document (Hash) of the data to be saved to MongoDB
905
- document = {
906
- :time => log.time,
907
- :host_name => SemanticLogger::Appender::MongoDB.host_name,
908
- :pid => $PID,
909
- :thread_name => log.thread_name,
910
- :name => log.name,
911
- :level => log.level,
912
- :level_index => log.level_index,
913
- }
914
- document[:application] = 'MyApplication'
915
- document[:message] = SemanticLogger::Appender::MongoDB.strip_colorizing(log.message) if log.message
916
- document[:duration] = log.duration if log.duration
917
- document[:tags] = log.tags if log.tags && (log.tags.size > 0)
918
- document[:payload] = log.payload if log.payload
919
- document[:exception] = {
920
- :name => log.exception.class.name,
921
- :message => log.exception.message,
922
- :stack_trace => log.exception.backtrace
923
- } if log.exception
924
- document
925
- end
926
- SemanticLogger.add_appender(mongodb_appender)
927
- ```
928
-
929
- ## Performance
930
-
931
- The traditional logging implementations write their log information to file in the
932
- same thread of execution as the program itself. This means that for every log entry
933
- the program has to wait for the data to be written.
934
-
935
- With Semantic Logger it uses a dedicated thread for logging so that writing to
936
- the log file or other appenders does not hold up program execution.
937
-
938
- Also, since the logging is in this separate thread there is no impact to program
939
- execution if we decided to add another appender.
940
- For example, log to both a file and a MongoDB collection.
941
-
942
- ## Log Rotation
943
-
944
- Since the log file is not re-opened with every call, when the log file needs
945
- to be rotated, use a copy-truncate operation over deleting the file.
946
-
947
- ## Why Semantic logging?
948
-
949
- Just as there is the initiative to add Semantic information to data on the web
950
- so that computers can directly understand the content without having to resort
951
- to complex regular expressions or machine learning techniques, it is important
952
- to be able to do the same with log files or data.
953
-
954
- Semantic Logger allows every log entry to have not only a message, but a payload
955
- that can be written to a file or a NOSQL destination.
956
-
957
- Once the logging data is in the NOSQL data store it can be queried quickly and
958
- efficiently. Some SQL data stores also allow complex data types that could be used
959
- for storing and querying the logging data
960
-
961
- Before writing SemanticLogger all of the following logging frameworks were thoroughly
962
- evaluated. None of them met the above Semantic requirements, or the performance requirements
963
- of hundreds of threads all logging at the same time:
964
- logback, logging, log4r, central_logger, whoops
965
-
966
- ## Architecture & Performance
967
-
968
- In order to ensure that logging does not hinder the performance of the application
969
- all log entries are written to thread-safe Queue. A separate thread is responsible
970
- for writing the log entries to each of the appenders.
971
-
972
- In this way formatting and disk or network write delays will not affect the
973
- performance of the application. Also adding more than one appender does not affect
974
- the runtime performance of the application.
975
-
976
- The logging thread is automatically started on initialization. When the program
977
- terminates it will call flush on each of the appenders.
978
-
979
- Calling SemanticLogger::Logger#flush will wait until all outstanding log messages
980
- have been written and flushed to their respective appenders before returning.
981
-
982
- ## Write your own Appender
983
-
984
- To write your own appender it should meet the following requirements:
985
-
986
- * Inherit from SemanticLogger::Base
987
- * In the initializer connect to the resource being logged to
988
- * Implement #log(log) which needs to write to the relevant resource
989
- * Implement #flush if the resource can be flushed
990
- * Write a test for the new appender
991
-
992
- The #log method takes the log struct as a parameter which is described above.
993
-
994
- Basic outline for an Appender:
995
-
996
- ```ruby
997
- require 'semantic_logger'
998
-
999
- class SimpleAppender < SemanticLogger::Appender::Base
1000
- def initialize(level=nil, &block)
1001
- # Set the log level and formatter if supplied
1002
- super(level, &block)
1003
- end
1004
-
1005
- # Display the log struct and the text formatted output
1006
- def log(log)
1007
- p log
1008
- puts formatter.call(log)
1009
- end
1010
-
1011
- # Optional
1012
- def flush
1013
- puts "Flush :)"
1014
- end
1015
- end
1016
- ```
1017
-
1018
- Sample program calling the above appender:
1019
- ```ruby
1020
- SemanticLogger.default_level = :trace
1021
- # Log to file dev.log
1022
- SemanticLogger.add_appender('dev.log')
1023
- # Also log the above sample appender
1024
- SemanticLogger.add_appender(SimpleAppender.new)
1025
-
1026
- logger = SemanticLogger['Hello']
1027
- logger.info "Hello World"
1028
- ```
1029
-
1030
- Look at the [existing appenders](https://github.com/reidmorrison/semantic_logger/tree/master/lib/semantic_logger/appender) for good examples
1031
-
1032
- To have your appender included in the standard list of appenders, submit it along
1033
- with complete working tests.
1034
- See the [MongoDB Appender Test](https://github.com/reidmorrison/semantic_logger/blob/master/test/appender_mongodb_test.rb) for an example.
10
+ For complete documentation see: http://reidmorrison.github.io/semantic_logger
1035
11
 
1036
12
  ## Dependencies
1037
13
 
@@ -1053,8 +29,7 @@ As of SemanticLogger V2.0 the Rails logging is no longer automatically replaced
1053
29
  when including SemanticLogger. Include the [rails_semantic_logger](http://github.com/reidmorrison/rails_semantic_logger)
1054
30
  gem to replace the Rails default logger with SemanticLogger
1055
31
 
1056
- Meta
1057
- ----
32
+ ## Meta
1058
33
 
1059
34
  * Code: `git clone git://github.com/reidmorrison/semantic_logger.git`
1060
35
  * Home: <https://github.com/reidmorrison/semantic_logger>
@@ -1063,18 +38,15 @@ Meta
1063
38
 
1064
39
  This project uses [Semantic Versioning](http://semver.org/).
1065
40
 
1066
- Author
1067
- -------
41
+ ## Author
1068
42
 
1069
43
  Reid Morrison :: reidmo@gmail.com :: @reidmorrison
1070
44
 
1071
- Contributors
1072
- ------------
45
+ ## Contributors
1073
46
 
1074
47
  Marc Bellingrath :: marrrc.b@gmail.com
1075
48
 
1076
- License
1077
- -------
49
+ ## License
1078
50
 
1079
51
  Copyright 2012, 2013, 2014 Reid Morrison
1080
52