timber 1.0.3 → 1.0.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 8580577f8c85abeb5eacec510136a2bf05d72406
-  data.tar.gz: 7a08aa6f00eda500d016c413ccf2b95d759b9266
+  metadata.gz: 08cb61746c20295dafbfcf1a098664dd9a7ac0ce
+  data.tar.gz: f7a59dd084e99faac34169d37bc49f33a0aafba0
 SHA512:
-  metadata.gz: b906867c8198c1d94052856b6e7b402af88bd252b770ed9a8a33aae7284a3f562569f4fde7202c759eb786e28f8b85b0b5a2b1ab8c2b5d63bde56f1ee686d404
-  data.tar.gz: d4afc694df19c1c90a4711b7d1762bb0f203cf4521aa44a1f63d5e7f66c8cef37258f7af2c8e6fb73b166305aa43a1b4bfce3a2ea0c625e1951e9bc0726657dc
+  metadata.gz: 40ffc77bceab9aec868f3f0592054970bd535abcdc6fda63101859605cd68a392faf2a7f7f13970cd6091370f2b711dc2f52072cede1f2d3aee760ef76274ba6
+  data.tar.gz: c2b7781da71ee771da0bb771295b83166f22446cf77bc27905d1a08311079b4b1a7ac3f43ca0011002148dff372f185c3180845905e8b8646960624278122759

data/README.md

@@ -11,34 +11,48 @@
 
 
 1. [What is timber?](#what-is-timber)
-1. [How does it work?](#what-is-timber)
-2. [Logging Custom Events](#logging-custom-events)
-3. [The Timber Console / Pricing](#the-timber-console-pricing)
-2. [Install](#install)
+2. [Why timber?](#why-timber)
+3. [How does it work?](#how-does-it-work)
+4. [Logging Custom Events](#logging-custom-events)
+5. [The Timber Console / Pricing](#the-timber-console-pricing)
+6. [Install](#install)
 
 
 ## What is Timber?
 
-Timber automatically structures your logs with events and context in a non-proprietary JSON format.
-It’s simple, quick, managed, and has absolutely no risk of code debt or lock-in.
-It’s just good ol’ logging.
+Glad you asked! :) Timber takes a different approach to logging, in that it automatically
+enriches and structures your logs without altering the essence of your original log messages.
+Giving you the best of both worlds: human readable logs *and* rich structured data.
 
-Timber’s philosophy is that application insight should be open and owned by you.
-And there is no better vehicle than logging:
+And it does so with absolutely no lock-in or risk of code debt. It's just good ol' loggin'™!
+For example:
 
-1. It’s a shared practice that has been around since the dawn of computers.
-2. It’s baked into every language, library, and framework. Even your own apps.
-3. The data is entirely owned by you.
+1. The resulting log format, by deafult, is a simple, non-proprietary, JSON structure.
+2. The [`Timber::Logger`](lib/timber/events) class extends `Logger`, and will never change or
+   extend the public API.
+3. Where you send your logs is entirely up to you, but we hope you'll check out
+   [timber.io](https://timber.io). We've built a beautiful, modern, and *fast* console specifically
+   for the strutured data captured here.
 
-The problem is that logs are messy, noisy, and hard to use. Timber solves this by being
-application aware, properly structuring your logs, and optionally providing a [fast, modern,
-and beautiful console](https://timber.io) -- allowing you to realize the power of
-your logs.
+
+## Why Timber?
+
+Timber’s philosophy is that application insight should be open and owned by you. It should not
+require a myriad of services to accomplish. And there is no better vehicle than logging:
+
+1. The log is immutable and complete. [It is the truth](http://files.timber.io/images/log-is-the-truth.png) :)
+2. It’s a shared practice that has been around since the dawn of computers.
+3. It’s baked into every language, library, and framework. Even your own apps!
+4. The data is open, accessible, and entirely owned by you. Yay!
+
+The problem is that logs are unstructured, noisy, and hard to use. `grep` can only take you so
+far! Timber solves this by properly structuring your logs, making them easy to search and
+visualize -- enabling you to sanely realize the power of your logs.
 
 
 ## How does it work?
 
-Glad you asked! :) Timber automatically structures your logs by taking advantage of public APIs.
+Timber automatically structures your logs by taking advantage of public APIs.
 
 For example, by subscribing to `ActiveSupport::Notifications`, Timber can automatically turn this:
 
@@ -76,12 +90,12 @@ Into this:
 ```
 
 It does the same for `http requests`, `sql queries`, `exceptions`, `template renderings`,
-and any other event your framework logs. (for a full list see `Timber::Events`)
+and any other event your framework logs. (for a full list see [`Timber::Events`](lib/timber/events))
 
 
 ## Logging Custom Events
 
-> Another service? More code debt? :*(
+> Another service? More lock-in? :*(
 
 Nope! Logging custom events is Just Logging™. Check it out:
 
@@ -101,41 +115,9 @@ end
 Logger.warn PaymentRejectedEvent.new("abcd1234", 100, "Card expired")
 ```
 
-(for more examples, see the `Timber::Logger` docs)
-
-No mention of Timber anywhere! In fact, this approach pushes things the opposite way. What if,
-as a result of structured logging, you could start decoupling other services from your application?
-
-Before:
-
-```
-               |---[HTTP]---> sentry / bugsnag / etc
-My Application |---[HTTP]---> librato / graphite / etc
-               |---[HTTP]---> new relic / etc
-               |--[STDOUT]--> logs
-                                |---> Logging service
-                                |---> S3
-                                |---> RedShift
-```
-
-
-After:
-
-```
-                                                    |-- sentry / bugsnag / etc
-                                                    |-- librato / graphite / etc
-My Application |--[STDOUT]--> logs ---> Timber ---> |-- new relic / etc
-                               ^                    |-- S3
-                               |                    |-- RedShift
-                               |                                 ^
-                    fast, efficient, durable,                    |
-                     replayable, auditable,        change any of these without
-                          just logging                  touching your code
-                                                       *and* backfill them!
-```
-
-[Mind-blown!](http://i.giphy.com/EldfH1VJdbrwY.gif)
+(for more examples, see [the `Timber::Logger` docs](lib/timber/logger.rb))
 
+No mention of Timber anywhere!
 
 
 ## The Timber Console / Pricing
@@ -186,6 +168,19 @@ after you add your application.
 
 *Other transport methods coming soon!*
 
+
+#### Rails TaggedLogging?
+
+No probs! Use it as normal, Timber will even pull out the tags and include them in the `context`.
+
+```ruby
+config.logger = ActiveSupport::TaggedLogging.new(Timber::Logger.new(STDOUT))
+```
+
+**Warning**: Tags lack meaningful descriptions, they are a poor mans context. Not to worry though!
+Timber provides a simple system for adding custom context that you can optionally use. Checkout
+[the `Timber::CurrentContext` docs](lib/timber/current_context.rb) for examples.
+
 ---
 
 That's it! Log to your heart's content.

data/lib/timber.rb

@@ -1,5 +1,6 @@
 # core classes
 require "json" # brings to_json to the core classes
+require "msgpack" # brings to_msgpack to the core classes
 
 # Base (must come first, order matters)
 require "timber/config"

data/lib/timber/context.rb

@@ -2,8 +2,14 @@ module Timber
   # Base class for all `Timber::Contexts::*` classes.
   # @private
   class Context
+    class << self
+      def keyspace
+        @keyspace || raise(NotImplementedError.new)
+      end
+    end
+
     def keyspace
-      raise NoImplementedError.new
+      self.class.keyspace
     end
 
     def as_json(options = {})

data/lib/timber/contexts.rb

@@ -1,6 +1,7 @@
 require "timber/contexts/custom"
 require "timber/contexts/http"
 require "timber/contexts/organization"
+require "timber/contexts/tags"
 require "timber/contexts/user"
 
 module Timber

data/lib/timber/contexts/custom.rb

@@ -8,6 +8,8 @@ module Timber
     #     # ... anything logged here will have the context ...
     #   end
     class Custom < Context
+      @keyspace = :custom
+
       attr_reader :type, :data
 
       def initialize(attributes)
@@ -15,10 +17,6 @@ module Timber
         @data = attributes[:data] || raise(ArgumentError.new(":data is required"))
       end
 
-      def keyspace
-        :custom
-      end
-
       def as_json(_options = {})
         {type => data}
       end

data/lib/timber/contexts/http.rb

@@ -7,6 +7,8 @@ module Timber
     # @note This context should be installed automatically through probes,
     #   such as the {Probes::RackHTTPContext} probe.
     class HTTP < Context
+      @keyspace = :http
+
       attr_reader :method, :path, :remote_addr, :request_id
 
       def initialize(attributes)
@@ -16,10 +18,6 @@ module Timber
         @request_id = attributes[:request_id]
       end
 
-      def keyspace
-        :http
-      end
-
       def as_json(_options = {})
         {:method => method, :path => path, :remote_addr => remote_addr, :request_id => request_id}
       end

data/lib/timber/contexts/organization.rb

@@ -16,6 +16,8 @@ module Timber
     #   end
     #
     class Organization < Context
+      @keyspace = :organization
+
       attr_reader :id, :name
 
       def initialize(attributes)
@@ -23,10 +25,6 @@ module Timber
         @name = attributes[:name]
       end
 
-      def keyspace
-        :organization
-      end
-
       def as_json(_options = {})
         {id: id, name: name}
       end

data/lib/timber/contexts/tags.rb

@@ -0,0 +1,22 @@
+module Timber
+  module Contexts
+    # Adds unnamed tags to the context.
+    #
+    # **Warning:** It is highly recommend that you use custom contexts instead. As they are
+    # more descriptive. This module exists primarily to support the ActiveSupport::TaggedLogging
+    # antipattern.
+    class Tags < Context
+      @keyspace = :tags
+
+      attr_reader :values
+
+      def initialize(attributes)
+        @values = attributes[:values] || raise(ArgumentError.new(":values is required"))
+      end
+
+      def as_json(_options = {})
+        values
+      end
+    end
+  end
+end

data/lib/timber/contexts/user.rb

@@ -17,6 +17,8 @@ module Timber
     #   end
     #
     class User < Context
+      @keyspace = :user
+
       attr_reader :id, :name
 
       def initialize(attributes)
@@ -24,10 +26,6 @@ module Timber
         @name = attributes[:name]
       end
 
-      def keyspace
-        :user
-      end
-
       def as_json(_options = {})
         {id: id, name: name}
       end

data/lib/timber/current_context.rb

@@ -4,40 +4,101 @@ module Timber
   # Holds the current context in a thread safe memory storage. This context is
   # appended to every log line. Think of context as join data between your log lines,
   # allowing you to relate them and filter them appropriately.
+  #
+  # @note Because context is appended to every log line, it is recommended that you limit this
+  #   to only neccessary data needed to relate your log lines.
   class CurrentContext
     include Singleton
 
     THREAD_NAMESPACE = :_timber_current_context.freeze
 
     class << self
-      # Convenience method for {#with}.
-      #
-      # @example Adding a context
-      #   custom_context = Timber::Contexts::Custom.new(type: :keyspace, data: %{my: "data"})
-      #   Timber::CurrentContext.with(custom_context) do
-      #     # ... anything logged here will have the context ...
-      #   end
+      # Convenience method for {#with}. See {#with} for full details and examples.
       def with(*args, &block)
         instance.with(*args, &block)
       end
+
+      # Convenience method for {#add}. See {#add} for full details and examples.
+      def add(*args)
+        instance.add(*args)
+      end
+
+      # Convenience method for {#remove}. See {#remove} for full details and examples.
+      def remove(*args)
+        instance.remove(*args)
+      end
+
+      def hash(*args)
+        instance.hash(*args)
+      end
     end
 
-    # Adds a context to the current stack.
-    def with(data)
-      key = data.keyspace
-      hash[key] = data
+    # Adds a context and then removes it when the block is finished executing.
+    #
+    # @note Because context is included with every log line, it is recommended that you limit this
+    #   to only neccessary data.
+    #
+    # @example Adding a custom context
+    #   custom_context = Timber::Contexts::Custom.new(type: :organization, data: %{id: 1, name: "Timber"})
+    #   Timber::CurrentContext.with(custom_context) do
+    #     # ... anything logged here will include the context ...
+    #   end
+    #   # Be sure to checkout Timber::Contexts! These are officially supported and many of these
+    #   # will be automatically included via Timber::Probes
+    #
+    # @example Adding multiple contexts
+    #   Timber::CurrentContext.with(context1, context2) { ... }
+    def with(*contexts)
+      add(*contexts)
       yield
     ensure
-      hash.delete(key)
+      contexts.each do |context|
+        if context.keyspace == :custom
+          # Custom contexts are merged and should be removed the same
+          hash[context.keyspace].delete(context.type)
+        else
+          remove(context)
+        end
+      end
     end
 
-    def snapshot
-      hash.clone
+    # Adds contexts but does not remove them. See {#with} for automatic maintenance and {#remove}
+    # to remove them yourself.
+    #
+    # @note Because context is included with every log line, it is recommended that you limit this
+    #   to only neccessary data.
+    def add(*contexts)
+      contexts.each do |context|
+        key = context.keyspace
+        json = context.as_json # Convert to json now so that we aren't doing it for every line
+        if key == :custom
+          # Custom contexts are merged into the space
+          hash[key] ||= {}
+          hash[key].merge(json)
+        else
+          hash[key] = json
+        end
+      end
     end
 
-    private
-      def hash
-        Thread.current[THREAD_NAMESPACE] ||= {}
+    # Removes a context. This must be a {Timber::Context} type. See {Timber::Contexts} for a list.
+    # If you wish to remove by key, or some other way, use {#hash} and modify the hash accordingly.
+    def remove(*contexts)
+      contexts.each do |context|
+        hash.delete(context.keyspace)
       end
+    end
+
+    # The internal hash that is maintained. It is recommended that you use {#with} and {#add}
+    # for hash maintenance.
+    def hash
+      Thread.current[THREAD_NAMESPACE] ||= {}
+    end
+
+    # Snapshots the current context so that you get a moment in time representation of the context,
+    # since the context can change as execution proceeds.
+    def snapshot
+      hash.clone
+    end
   end
 end

data/lib/timber/log_devices/http.rb

@@ -1,19 +1,17 @@
-require "monitor"
-require "msgpack"
+require "timber/log_devices/http/triggered_buffer"
 
 module Timber
   module LogDevices
-    # A log device that buffers and sends logs to the Timber API over HTTP in intervals. The buffer
-    # uses MessagePack::Buffer, which is fast, efficient with memory, and reduces
-    # the payload size sent to Timber.
+    # A log device that buffers and delivers log messages over HTTPS to the Timber API in batches.
+    # The buffer and delivery strategy are very efficient and the log messages will be delivered in
+    # msgpack format.
+    #
+    # See {#initialize} for options and more details.
     class HTTP
-      class DeliveryError < StandardError; end
-
       API_URI = URI.parse("https://api.timber.io/http_frames")
-      CONTENT_TYPE = "application/json".freeze
+      CONTENT_TYPE = "application/x-timber-msgpack-frame-1".freeze
       CONNECTION_HEADER = "keep-alive".freeze
       USER_AGENT = "Timber Ruby Gem/#{Timber::VERSION}".freeze
-
       HTTPS = Net::HTTP.new(API_URI.host, API_URI.port).tap do |https|
         https.use_ssl = true
         https.read_timeout = 30
@@ -23,60 +21,100 @@ module Timber
         end
         https.open_timeout = 10
       end
+      DELIVERY_FREQUENCY_SECONDS = 2.freeze
+      RETRY_LIMIT = 3.freeze
+      BACKOFF_RATE_SECONDS = 3.freeze
 
-      DEFAULT_DELIVERY_FREQUENCY = 2.freeze
 
-      # Instantiates a new HTTP log device.
+      # Instantiates a new HTTP log device that can be passed to {Timber::Logger#initialize}.
       #
       # @param api_key [String] The API key provided to you after you add your application to
       #   [Timber](https://timber.io).
       # @param [Hash] options the options to create a HTTP log device with.
-      # @option attributes [Symbol] :frequency_seconds (2) How often the client should
+      # @option attributes [Symbol] :payload_limit_bytes Determines the maximum size in bytes that
+      #   and HTTP payload can be. Please see {TriggereBuffer#initialize} for the default.
+      # @option attributes [Symbol] :buffer_limit_bytes Determines the maximum size of the total
+      #   buffer. This should be many times larger than the `:payload_limit_bytes`.
+      #   Please see {TriggereBuffer#initialize} for the default.
+      # @option attributes [Symbol] :buffer_overflow_handler (nil) When a single message exceeds
+      #   `:payload_limit_bytes` or the entire buffer exceeds `:buffer_limit_bytes`, the Proc
+      #   passed to this option will be called with the msg that would overflow the buffer. See
+      #   the examples on how to use this properly.
+      # @option attributes [Symbol] :delivery_frequency_seconds (2) How often the client should
       #   attempt to deliver logs to the Timber API. The HTTP client buffers logs between calls.
+      #
+      # @example Basic usage
+      #   Timber::Logger.new(Timber::LogDevices::HTTP.new("my_timber_api_key"))
+      #
+      # @example Handling buffer overflows
+      #   # Persist overflowed lines to a file
+      #   # Note: You could write these to any permanent storage.
+      #   overflow_log_path = "/path/to/my/overflow_log.log"
+      #   overflow_handler = Proc.new { |log_line_msg| File.write(overflow_log_path, log_line_ms) }
+      #   http_log_device = Timber::LogDevices::HTTP.new("my_timber_api_key",
+      #     buffer_overflow_handler: overflow_handler)
+      #   Timber::Logger.new(http_log_device)
       def initialize(api_key, options = {})
         @api_key = api_key
-        @buffer = []
-        @monitor = Monitor.new
-        @delivery_thread = Thread.new do
-          at_exit { deliver }
+        @buffer = TriggeredBuffer.new(
+          payload_limit_bytes: options[:payload_limit_bytes],
+          limit_bytes: options[:buffer_limit_bytes],
+          overflow_handler: options[:buffer_overflow_handler]
+        )
+        @delivery_interval_thread = Thread.new do
           loop do
-            sleep options[:frequency_seconds] || DEFAULT_DELIVERY_FREQUENCY
-            deliver
+            sleep(options[:delivery_frequency_seconds] || DELIVERY_FREQUENCY_SECONDS)
+            buffer_for_delivery = @buffer.reserve
+            if buffer_for_delivery
+              deliver(buffer_for_delivery)
+            end
           end
         end
       end
 
+      # Write a new log line message to the buffer, and deliver if the msg exceeds the
+      # payload limit.
       def write(msg)
-        @monitor.synchronize {
-          @buffer << msg
-        }
+        buffer_for_delivery = @buffer.write(msg)
+        if buffer_for_delivery
+          deliver(buffer_for_delivery)
+        end
+        true
       end
 
+      # Closes the log device, cleans up, and attempts one last delivery.
       def close
-        @delivery_thread.kill
+        @delivery_interval_thread.kill
+        buffer_for_delivery = @buffer.reserve
+        if buffer_for_delivery
+          deliver(buffer_for_delivery)
+        end
       end
 
       private
-        def deliver
-          body = @buffer.read
+        def deliver(body)
+          Thread.new do
+            RETRY_LIMIT.times do |try_index|
+              request = Net::HTTP::Post.new(API_URI.request_uri).tap do |req|
+                req['Authorization'] = authorization_payload
+                req['Connection'] = CONNECTION_HEADER
+                req['Content-Type'] = CONTENT_TYPE
+                req['User-Agent'] = USER_AGENT
+                req.body = body
+              end
 
-          request = Net::HTTP::Post.new(API_URI.request_uri).tap do |req|
-            req['Authorization'] = authorization_payload
-            req['Connection'] = CONNECTION_HEADER
-            req['Content-Type'] = CONTENT_TYPE
-            req['User-Agent'] = USER_AGENT
-            req.body = body
-          end
-
-          HTTPS.request(request).tap do |res|
-            code = res.code.to_i
-            if code < 200 || code >= 300
-              raise DeliveryError.new("Bad response from Timber API - #{res.code}: #{res.body}")
+              res = HTTPS.request(request)
+              code = res.code.to_i
+              if code < 200 || code >= 300
+                Config.instance.logger.debug("Timber HTTP delivery failed - #{res.code}: #{res.body}")
+                sleep((try_index + 1) * BACKOFF_RATE_SECONDS)
+              else
+                @buffer.remove(body)
+                Config.instance.logger.debug("Timber HTTP delivery successful - #{code}")
+                break # exit the loop
+              end
             end
-            Config.instance.logger.debug("Success! #{code}: #{res.body}")
           end
-
-          @buffer.clear
         end
 
         def authorization_payload

data/lib/timber/log_devices/http/triggered_buffer.rb

@@ -0,0 +1,79 @@
+require "monitor"
+
+module Timber
+  module LogDevices
+    class HTTP
+      # Maintains a triggered buffer, where the trigger is {PAYLOAD_LIMIT_BYTES}. Once the buffer
+      # exceeds this limit it will lock and return that buffer up to that point while still making
+      # a new buffer available for writes. This ensures that the HTTP client can attempt to deliver
+      # the buffer contents without blocking execution of the application.
+      #
+      # If the overall buffer exceeeds the overall limit (specified by the `:limit_bytes` option),
+      # then a buffer overflow is triggered. This can be customized using the `:overflow_handler`
+      # option.
+      class TriggeredBuffer
+        DEFAULT_PAYLOAD_LIMIT_BYTES = 5_000_000 # 5mb, the Timber API will not accept messages larger than this
+        DEFAULT_LIMIT_BYTES = 50_000_000 # 50mb
+
+        def initialize(options = {})
+          @buffers = []
+          @monitor = Monitor.new
+          @payload_limit_bytes = options[:payload_limit_bytes] || DEFAULT_PAYLOAD_LIMIT_BYTES
+          @limit_bytes = options[:limit_bytes] || DEFAULT_LIMIT_BYTES
+          @overflow_handler = options[:overflow_handler]
+        end
+
+        def write(msg)
+          if msg.bytesize > @payload_limit_bytes || (msg.bytesize + total_bytesize) > @limit_bytes
+            handle_overflow(msg)
+            return nil
+          end
+
+          @monitor.synchronize do
+            buffer = writable_buffer
+            if @buffers == [] || buffer.nil? || buffer.frozen?
+              @buffers << msg
+              nil
+            elsif (buffer.bytesize + msg.bytesize) > @payload_limit_bytes
+              @buffers << msg
+              buffer.freeze
+            else
+              buffer << msg
+              nil
+            end
+          end
+        end
+
+        def reserve
+          @monitor.synchronize do
+            buffer = writable_buffer
+            if buffer
+              buffer.freeze
+            end
+          end
+        end
+
+        def remove(buffer)
+          @monitor.synchronize do
+            @buffers.delete(buffer)
+          end
+        end
+
+        private
+          def total_bytesize
+            @buffers.reduce(0) { |acc, buffer| acc + buffer.bytesize }
+          end
+
+          def writable_buffer
+            @buffers.find { |buffer| !buffer.frozen? }
+          end
+
+          def handle_overflow(msg)
+            if @overflow_handler
+              @overflow_handler.call(msg)
+            end
+          end
+      end
+    end
+  end
+end

data/lib/timber/logger.rb

@@ -81,6 +81,27 @@ module Timber
         end
     end
 
+    # Structures your log messages into Timber's hybrid format, which makes
+    # it easy to read while also appending the appropriate metadata.
+    #
+    #   logger = Timber::Logger.new(STDOUT)
+    #   logger.formatter = Timber::JSONFormatter.new
+    #
+    # Example message:
+    #
+    #   My log message @timber.io {"level":"info","dt":"2016-09-01T07:00:00.000000-05:00"}
+    #
+    class HybridFormatter < Formatter
+      METADATA_CALLOUT = "@timber.io".freeze
+
+      def call(severity, time, progname, msg)
+        log_entry = build_log_entry(severity, time, progname, msg)
+        metadata = log_entry.to_json(:except => [:message])
+        # use << for concatenation for performance reasons
+        log_entry.message.gsub("\n", "\\n") << " " << METADATA_CALLOUT << " " << metadata << "\n"
+      end
+    end
+
     # Structures your log messages into JSON.
     #
     #   logger = Timber::Logger.new(STDOUT)
@@ -97,24 +118,19 @@ module Timber
       end
     end
 
-    # Structures your log messages into Timber's hybrid format, which makes
-    # it easy to read while also appending the appropriate metadata.
+    # Structures your log messages into JSON.
     #
     #   logger = Timber::Logger.new(STDOUT)
     #   logger.formatter = Timber::JSONFormatter.new
     #
     # Example message:
     #
-    #   My log message @timber.io {"level":"info","dt":"2016-09-01T07:00:00.000000-05:00"}
+    #   {"level":"info","dt":"2016-09-01T07:00:00.000000-05:00","message":"My log message"}
     #
-    class HybridFormatter < Formatter
-      METADATA_CALLOUT = "@timber.io".freeze
-
+    class MsgPackFormatter < Formatter
       def call(severity, time, progname, msg)
-        log_entry = build_log_entry(severity, time, progname, msg)
-        metadata = log_entry.to_json(:except => [:message])
         # use << for concatenation for performance reasons
-        log_entry.message.gsub("\n", "\\n") << " " << METADATA_CALLOUT << " " << metadata << "\n"
+        build_log_entry(severity, time, progname, msg).as_json.to_msgpack << "\n"
       end
     end
 
@@ -127,7 +143,20 @@ module Timber
     #   logger.formatter = Timber::Logger::JSONFormatter.new
     def initialize(*args)
       super(*args)
-      self.formatter = HybridFormatter.new
+      if args.size == 1 and args.first.is_a?(LogDevices::HTTP)
+        self.formatter = MsgPackFormatter.new
+      else
+        self.formatter = HybridFormatter.new
+      end
+    end
+
+    def formatter=(value)
+      if @dev.is_a?(Timber::LogDevices::HTTP)
+        raise ArgumentError.new("The formatter cannot be changed when using the " +
+          "Timber::LogDevices::HTTP log device. The MsgPackFormatter must be used for proper " +
+          "delivery.")
+      end
+      super
     end
 
     # Backwards compatibility with older ActiveSupport::Logger versions

data/lib/timber/probes.rb

@@ -2,6 +2,7 @@ require "timber/probes/action_controller_log_subscriber"
 require "timber/probes/action_dispatch_debug_exceptions"
 require "timber/probes/action_view_log_subscriber"
 require "timber/probes/active_record_log_subscriber"
+require "timber/probes/active_support_tagged_logging"
 require "timber/probes/rack_http_context"
 require "timber/probes/rails_rack_logger"
 
@@ -14,6 +15,7 @@ module Timber
       ActionDispatchDebugExceptions.insert!
       ActionViewLogSubscriber.insert!
       ActiveRecordLogSubscriber.insert!
+      ActiveSupportTaggedLogging.insert!
       RackHTTPContext.insert!(middleware, insert_before)
       RailsRackLogger.insert!
     end

data/lib/timber/probes/active_support_tagged_logging.rb

@@ -0,0 +1,106 @@
+module Timber
+  module Probes
+    # Reponsible for automatimcally tracking SQL query events in `ActiveRecord`, while still
+    # preserving the default log style.
+    class ActiveSupportTaggedLogging < Probe
+      module FormatterMethods
+        def self.included(mod)
+          mod.module_eval do
+            alias_method :_timber_original_push_tags, :push_tags
+            alias_method :_timber_original_pop_tags, :pop_tags
+
+            def call(severity, timestamp, progname, msg)
+              if is_a?(Timber::Logger::Formatter)
+                # Don't convert the message into a string
+                super(severity, timestamp, progname, msg)
+              else
+                super(severity, timestamp, progname, "#{tags_text}#{msg}")
+              end
+            end
+
+            def push_tags(*tags)
+              _timber_original_push_tags(*tags).tap do
+                if current_tags.size > 0
+                  context = Contexts::Tags.new(values: current_tags)
+                  CurrentContext.add(context)
+                end
+              end
+            end
+
+            def pop_tags(size = 1)
+              _timber_original_pop_tags(size).tap do
+                if current_tags.size == 0
+                  CurrentContext.remove(Contexts::Tags)
+                else
+                  context = Contexts::Tags.new(values: current_tags)
+                  CurrentContext.add(context)
+                end
+              end
+            end
+          end
+        end
+      end
+
+      module LoggerMethods
+        def self.included(klass)
+          klass.class_eval do
+            alias_method :_timber_original_push_tags, :push_tags
+            alias_method :_timber_original_pop_tags, :pop_tags
+
+            def push_tags(*tags)
+              _timber_original_push_tags(*tags).tap do
+                if current_tags.size > 0
+                  context = Contexts::Tags.new(values: current_tags)
+                  CurrentContext.add(context)
+                end
+              end
+            end
+
+            def pop_tags(size = 1)
+              _timber_original_pop_tags(size).tap do
+                if current_tags.size == 0
+                  CurrentContext.remove(Contexts::Tags)
+                else
+                  context = Contexts::Tags.new(values: current_tags)
+                  CurrentContext.add(context)
+                end
+              end
+            end
+
+            def add(severity, message = nil, progname = nil, &block)
+              if message.nil?
+                if block_given?
+                  message = block.call
+                else
+                  message = progname
+                  progname = nil #No instance variable for this like Logger
+                end
+              end
+              if @logger.is_a?(Timber::Logger)
+                @logger.add(severity, message, progname)
+              else
+                @logger.add(severity, "#{tags_text}#{message}", progname)
+              end
+            end
+          end
+        end
+      end
+
+      def initialize
+        require "active_support/tagged_logging"
+      rescue LoadError => e
+        raise RequirementNotMetError.new(e.message)
+      end
+
+      def insert!
+        if defined?(ActiveSupport::TaggedLogging::Formatter)
+          return true if ActiveSupport::TaggedLogging::Formatter.include?(FormatterMethods)
+          ActiveSupport::TaggedLogging::Formatter.send(:include, FormatterMethods)
+        else
+          return true if ActiveSupport::TaggedLogging.include?(LoggerMethods)
+          ActiveSupport::TaggedLogging.send(:include, LoggerMethods)
+        end
+      end
+    end
+  end
+end

data/lib/timber/probes/rack_http_context.rb

@@ -15,7 +15,7 @@ module Timber
             remote_addr: request.ip,
             request_id: request_id(env)
           )
-          CurrentContext.instance.with(context) do
+          CurrentContext.with(context) do
             @app.call(env)
           end
         end

data/lib/timber/version.rb

@@ -1,3 +1,3 @@
 module Timber
-  VERSION = "1.0.3"
+  VERSION = "1.0.4"
 end

data/spec/timber/log_devices/http/triggered_buffer_spec.rb

@@ -0,0 +1,58 @@
+require "spec_helper"
+
+describe Timber::LogDevices::HTTP::TriggeredBuffer do
+  describe "#write" do
+    it "should trigger a buffer overflow for large messages" do
+      buffer = described_class.new(:payload_limit_bytes => 10)
+      msg = "a" * 11
+      expect(buffer).to receive(:handle_overflow).exactly(1).times.with(msg)
+      buffer.write(msg)
+    end
+
+    it "should trigger a buffer overflow when exceeding the limit" do
+      buffer = described_class.new(:limit_bytes => 10)
+      msg = "a" * 11
+      expect(buffer).to receive(:handle_overflow).exactly(1).times.with(msg)
+      buffer.write(msg)
+    end
+
+    it "should start a new buffer when empty and append when not" do
+      buffer = described_class.new
+      result = buffer.write("test")
+      expect(result).to be_nil
+      expect(buffer.send(:writable_buffer)).to eq("test")
+      result = buffer.write("again")
+      expect(result).to be_nil
+      expect(buffer.send(:writable_buffer)).to eq("testagain")
+    end
+
+    it "should return the old buffer when it has exceeded it's limit" do
+      buffer = described_class.new(:payload_limit_bytes => 10)
+      msg = "a" * 6
+      result = buffer.write(msg)
+      expect(result).to be_nil
+      result = buffer.write(msg)
+      expect(result).to eq(msg)
+      expect(result).to be_frozen
+    end
+
+    it "should write a new buffer when the latest is frozen" do
+      buffer = described_class.new
+      buffer.write("test")
+      result = buffer.reserve
+      expect(result).to eq("test")
+      buffer.write("again")
+      expect(buffer.send(:writable_buffer)).to eq("again")
+    end
+  end
+
+  describe "#reserve" do
+    it "should reserve the latest buffer and freeze it" do
+      buffer = described_class.new
+      buffer.write("test")
+      result = buffer.reserve
+      expect(result).to eq("test")
+      expect(result).to be_frozen
+    end
+  end
+end

data/spec/timber/log_devices/http_spec.rb

@@ -1,62 +1,85 @@
-# require "spec_helper"
-
-# describe Timber::LogDevices::HTTP do
-#   # We have to define our own at_exit method, because the mocks and
-#   # everything are stripped out before. Otherwise it tries to issue
-#   # a request :(
-#   before(:each) do
-#     described_class.class_eval do
-#       def at_exit; true; end
-#     end
-#   end
-
-#   describe "#initialize" do
-#     it "should start a thread for delivery" do
-#       allow_any_instance_of(described_class).to receive(:at_exit).exactly(1).times.and_return(true)
-#       expect_any_instance_of(described_class).to receive(:deliver).exactly(2).times.and_return(true)
-#       http = described_class.new("MYKEY", frequency_seconds: 0.1)
-#       thread = http.instance_variable_get(:@delivery_thread)
-#       expect(thread).to be_alive
-#       sleep 0.25 # allow 2 iterations
-#       http.close
-#     end
-#   end
-
-#   describe "#write" do
-#     let(:http) { described_class.new("MYKEY") }
-#     let(:buffer) { http.instance_variable_get(:@buffer) }
-
-#     after(:each) { http.close }
-
-#     it "should buffer the messages" do
-#       http.write("test log message")
-#       expect(buffer.read).to eq("test log message")
-#     end
-#   end
-
-#   describe "#deliver" do
-#     let(:http) { described_class.new("MYKEY") }
-#     let(:buffer) { http.instance_variable_get(:@buffer) }
-
-#     after(:each) { http.close }
-
-#     it "should delivery properly and flush the buffer" do
-#       expect_any_instance_of(described_class).to receive(:at_exit).exactly(1).times.and_return(true)
-#       stub = stub_request(:post, "https://api.timber.io/http_frames").
-#         with(
-#           :body => "test log message",
-#           :headers => {'Authorization'=>'Basic TVlLRVk=', 'Connection'=>'keep-alive', 'Content-Type'=>'application/json', 'User-Agent'=>'Timber Ruby Gem/1.0.0'}
-#         ).
-#         to_return(:status => 200, :body => "", :headers => {})
-
-#       http.write("test log message")
-
-#       expect(buffer).to_not be_empty
-
-#       http.send(:deliver)
-
-#       expect(stub).to have_been_requested.times(1)
-#       expect(buffer).to be_empty
-#     end
-#   end
-# end
+require "spec_helper"
+
+describe Timber::LogDevices::HTTP do
+  describe "#initialize" do
+    it "should start a thread for delivery" do
+      expect_any_instance_of(described_class).to receive(:deliver).at_least(1).times.and_return(true)
+      http = described_class.new("MYKEY", delivery_frequency_seconds: 0.1)
+      thread = http.instance_variable_get(:@delivery_interval_thread)
+      expect(thread).to be_alive
+
+      http.write("my log message")
+      sleep 0.3 # too fast!
+    end
+  end
+
+  describe "#write" do
+    let(:http) { described_class.new("MYKEY") }
+    let(:buffer) { http.instance_variable_get(:@buffer) }
+
+    it "should buffer the messages" do
+      http.write("test log message")
+      expect(buffer.reserve).to eq("test log message")
+    end
+
+    context "with a low payload limit" do
+      let(:http) { described_class.new("MYKEY", :payload_limit_bytes => 20) }
+
+      it "should attempt a delivery when the payload limit is exceeded" do
+        message = "a" * 19
+        http.write(message)
+        expect(http).to receive(:deliver).exactly(1).times.with(message)
+        http.write("my log message")
+      end
+    end
+  end
+
+  describe "#close" do
+    let(:http) { described_class.new("MYKEY") }
+
+    it "should kill the delivery thread the messages" do
+      http.close
+      thread = http.instance_variable_get(:@delivery_interval_thread)
+      sleep 0.1 # too fast!
+      expect(thread).to_not be_alive
+    end
+
+    it "should attempt a delivery" do
+      message = "a" * 19
+      http.write(message)
+      expect(http).to receive(:deliver).exactly(1).times.with(message)
+      http.close
+    end
+  end
+
+  describe "#deliver" do
+    let(:http) { described_class.new("MYKEY") }
+
+    after(:each) { http.close }
+
+    it "should delivery properly and flush the buffer" do
+      stub = stub_request(:post, "https://api.timber.io/http_frames").
+        with(
+          :body => "test log message",
+          :headers => {
+            'Authorization' => 'Basic TVlLRVk=',
+            'Connection' => 'keep-alive',
+            'Content-Type' => 'application/x-timber-msgpack-frame-1',
+            'User-Agent' => "Timber Ruby Gem/#{Timber::VERSION}"
+          }
+        ).
+        to_return(:status => 200, :body => "", :headers => {})
+
+      http.write("test log message")
+      buffer = http.instance_variable_get(:@buffer)
+      buffers = buffer.instance_variable_get(:@buffers)
+      expect(buffers.size).to eq(1)
+      body = buffer.reserve
+      thread = http.send(:deliver, body)
+      thread.join
+
+      expect(stub).to have_been_requested.times(1)
+      expect(buffers.size).to eq(0)
+    end
+  end
+end

data/spec/timber/logger_spec.rb

@@ -29,7 +29,7 @@ describe Timber::Logger, :rails_23 => true do
         end
 
         around(:each) do |example|
-          Timber::CurrentContext.instance.with(http_context) do
+          Timber::CurrentContext.with(http_context) do
             example.run
           end
         end
@@ -76,13 +76,25 @@ describe Timber::Logger, :rails_23 => true do
       end
     end
 
-    context "with TaggedLogging" do
-      let(:logger) { ActiveSupport::TaggedLogging.new(Timber::Logger.new(io)) }
+    if defined?(ActiveSupport::TaggedLogging)
+      context "with TaggedLogging", :rails_23 => false do
+        let(:logger) { ActiveSupport::TaggedLogging.new(Timber::Logger.new(io)) }
 
-      it "should format properly with events" do
-        message = Timber::Events::SQLQuery.new(sql: "select * from users", time_ms: 56, message: "select * from users")
-        logger.info(message)
-        expect(io.string).to eq("select * from users @timber.io {\"level\":\"info\",\"dt\":\"2016-09-01T12:00:00.000000Z\",\"event\":{\"sql_query\":{\"sql\":\"select * from users\",\"time_ms\":56}}}\n")
+        it "should format properly with events" do
+          message = Timber::Events::SQLQuery.new(sql: "select * from users", time_ms: 56, message: "select * from users")
+          logger.tagged("tag") do
+            logger.info(message)
+          end
+          expect(io.string).to eq("select * from users @timber.io {\"level\":\"info\",\"dt\":\"2016-09-01T12:00:00.000000Z\",\"event\":{\"sql_query\":{\"sql\":\"select * from users\",\"time_ms\":56}},\"context\":{\"tags\":[\"tag\"]}}\n")
+        end
+      end
+    end
+
+    context "with the HTTP log device" do
+      let(:io) { Timber::LogDevices::HTTP.new("my_key") }
+
+      it "should use the msgpack formatter" do
+        expect(logger.formatter).to be_kind_of(Timber::Logger::MsgPackFormatter)
       end
     end
   end

data/spec/timber/probes/rack_http_context_spec.rb

@@ -41,11 +41,7 @@ describe Timber::Probes::RackHTTPContext do
         dispatch_rails_request("/rack_http")
         http_context = Thread.current[:_timber_context][:http]
 
-        expect(http_context).to be_kind_of(Timber::Contexts::HTTP)
-        expect(http_context.method).to eq("GET")
-        expect(http_context.path).to eq("/rack_http")
-        expect(http_context.remote_addr).to eq("123.456.789.10")
-        expect(http_context.request_id).to_not be_nil
+        expect(http_context).to eq({:method=>"GET", :path=>"/rack_http", :remote_addr=>"123.456.789.10", :request_id=>"unique-request-id-1234"})
         message = "Processing by RackHttpController#index as HTML @timber.io {\"level\":\"info\",\"dt\":\"2016-09-01T12:00:00.000000Z\",\"event\":{\"controller_call\":{\"controller\":\"RackHttpController\",\"action\":\"index\"}},\"context\":{\"http\":{\"method\":\"GET\",\"path\":\"/rack_http\",\"remote_addr\":\"123.456.789.10\",\"request_id\":\"unique-request-id-1234\"}}}\nCompleted 200 OK in 0.0ms (Views: 1.0ms) @timber.io {\"level\":\"info\",\"dt\":\"2016-09-01T12:00:00.000000Z\",\"event\":{\"http_response\":{\"status\":200,\"time_ms\":0.0}},\"context\":{\"http\":{\"method\":\"GET\",\"path\":\"/rack_http\",\"remote_addr\":\"123.456.789.10\",\"request_id\":\"unique-request-id-1234\"}}}\n"
         expect(io.string).to eq(message)
       end

data/timber.gemspec

@@ -8,8 +8,8 @@ Gem::Specification.new do |s|
   s.platform    = Gem::Platform::RUBY
   s.authors     = ["Timber Technologies, Inc."]
   s.email       = ["hi@timber.io"]
-  s.homepage    = "http://timber.io"
-  s.summary     = "Logs you'll actually use."
+  s.homepage    = "https://github.com/timberio/timber-ruby"
+  s.summary     = "Instant log gratification."
 
   s.required_ruby_version     = '>= 1.9.0'
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: timber
 version: !ruby/object:Gem::Version
-  version: 1.0.3
+  version: 1.0.4
 platform: ruby
 authors:
 - Timber Technologies, Inc.
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-12-14 00:00:00.000000000 Z
+date: 2016-12-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: msgpack
@@ -46,6 +46,7 @@ files:
 - lib/timber/contexts/custom.rb
 - lib/timber/contexts/http.rb
 - lib/timber/contexts/organization.rb
+- lib/timber/contexts/tags.rb
 - lib/timber/contexts/user.rb
 - lib/timber/current_context.rb
 - lib/timber/event.rb
@@ -61,6 +62,7 @@ files:
 - lib/timber/frameworks/rails.rb
 - lib/timber/log_devices.rb
 - lib/timber/log_devices/http.rb
+- lib/timber/log_devices/http/triggered_buffer.rb
 - lib/timber/log_entry.rb
 - lib/timber/logger.rb
 - lib/timber/probe.rb
@@ -72,6 +74,7 @@ files:
 - lib/timber/probes/action_view_log_subscriber/log_subscriber.rb
 - lib/timber/probes/active_record_log_subscriber.rb
 - lib/timber/probes/active_record_log_subscriber/log_subscriber.rb
+- lib/timber/probes/active_support_tagged_logging.rb
 - lib/timber/probes/rack_http_context.rb
 - lib/timber/probes/rails_rack_logger.rb
 - lib/timber/util.rb
@@ -92,6 +95,7 @@ files:
 - spec/support/timecop.rb
 - spec/support/webmock.rb
 - spec/timber/events_spec.rb
+- spec/timber/log_devices/http/triggered_buffer_spec.rb
 - spec/timber/log_devices/http_spec.rb
 - spec/timber/logger_spec.rb
 - spec/timber/probes/action_controller_log_subscriber_spec.rb
@@ -101,7 +105,7 @@ files:
 - spec/timber/probes/rack_http_context_spec.rb
 - spec/timber/probes/rails_rack_logger_spec.rb
 - timber.gemspec
-homepage: http://timber.io
+homepage: https://github.com/timberio/timber-ruby
 licenses: []
 metadata: {}
 post_install_message: 
@@ -123,7 +127,7 @@ rubyforge_project:
 rubygems_version: 2.6.8
 signing_key: 
 specification_version: 4
-summary: Logs you'll actually use.
+summary: Instant log gratification.
 test_files:
 - spec/spec_helper.rb
 - spec/support/action_controller.rb
@@ -139,6 +143,7 @@ test_files:
 - spec/support/timecop.rb
 - spec/support/webmock.rb
 - spec/timber/events_spec.rb
+- spec/timber/log_devices/http/triggered_buffer_spec.rb
 - spec/timber/log_devices/http_spec.rb
 - spec/timber/logger_spec.rb
 - spec/timber/probes/action_controller_log_subscriber_spec.rb