logstash-logger 0.5.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e6696816384e341e562b5cf39921a1f8d71accb0
-  data.tar.gz: a510f56283c8a70101dd5604efa2b5470edb5f5f
+  metadata.gz: 6ce22d1b662dce1b8438a6760bc747a141e810b8
+  data.tar.gz: ec33c1bf4b20ebeb54271d5b307d77c6f3195c35
 SHA512:
-  metadata.gz: a9b42b29215905770c72c3ad7ada65df908302a24e0389d636e0e7c3dbef631c06dfd57a304180adc5acdf3138ed81bb47ed9e8b465393a7c3130be107390892
-  data.tar.gz: ce2711e17ef4ce503da02e1e9561f55b61525d4766bfefda2ba5a794418e990116351b3c57f25939e0bd9be58242cf3e5b68096b72dfeb5576e3267d890df1cc
+  metadata.gz: 8832f5370c049bfaa6ebe74f91ba716c42e43ad0d2b1d31a6fc2c0379005f18d9a0a2842271af50958c39488671d38986e271683df8c6d684c6a3077f6c2bfb9
+  data.tar.gz: fa9188fd5ff3e1e3585c112bb35258cd4dce9a141e208b3c6123dc91a5db54f0d0c54d682ced533d4f3516090d97666245ec9b301c978f0d4756526f4233c8bb

data/CHANGELOG.md

@@ -1,3 +1,10 @@
+## 0.6.0
+- Support for logging to a file.
+- Support for logging to a Redis list.
+- Support for logging to a local Unix socket.
+- Railtie supports file logger, using default log path and `config.autoflush_log` configuration.
+- All `LogStashLogger` types now support a `sync` option, which controls if each message is automatically flushed.
+
 ## 0.5.0
 - Support for tagged logging. The interface was extracted from `ActiveSupport::TaggedLogging`
 and outputs to the `tags` key. (Thanks [pctj101](https://github.com/pctj101)!)

data/README.md

@@ -7,10 +7,12 @@ writing to a file or syslog since logstash can receive the structured data direc
 
 ## Features
 
-* Writes directly to logstash over a UDP or TCP connection.
+* Can write directly to logstash over a UDP or TCP/SSL connection.
+* Can write to a file, Redis, a unix socket, or stdout.
 * Always writes in logstash JSON format.
-* Logger can take a string message, a hash, a LogStash::Event, or a logstash-formatted json string as input.
+* Logger can take a string message, a hash, a `LogStash::Event`, or a JSON string as input.
 * Events are automatically populated with message, timestamp, host, and severity.
+* Easily integrates with Rails via configuration.
 
 ## Installation
 
@@ -28,11 +30,6 @@ Or install it yourself as:
 
 ## Basic Usage
 
-First set up a logstash agent to receive input over a UDP or TCP port.
-Then in ruby, create a `LogStashLogger` that writes to that port.
-
-The logger accepts a string message, a JSON string, a hash, or a `LogStash::Event`.
-
 ```ruby
 require 'logstash-logger'
 
@@ -40,8 +37,13 @@ require 'logstash-logger'
 logger = LogStashLogger.new(port: 5228)
 
 # Specify host and type (UDP or TCP) explicitly
-udp_logger = LogStashLogger.new(host: 'localhost', port: 5228, type: :udp)
-tcp_logger = LogStashLogger.new(host: 'localhost', port: 5229, type: :tcp)
+udp_logger = LogStashLogger.new(type: :udp, host: 'localhost', port: 5228)
+tcp_logger = LogStashLogger.new(type: :tcp, host: 'localhost', port: 5229)
+
+# Other types of loggers
+file_logger = LogStashLogger.new(type: :file, path: 'log/development.log', sync: true)
+unix_logger = LogStashLogger.new(type: :unix, path: '/tmp/sock')
+redis_logger = LogStashLogger.new(type: :redis)
 stdout_logger = LogStashLogger.new(type: :stdout)
 
 # The following messages are written to UDP port 5228:
@@ -65,8 +67,9 @@ logger.tagged('foo') { logger.fatal('bar') }
 
 ## Logstash Configuration
 
-In order for Logstash to correctly receive and parse the event, you will need to
-configure and run a UDP listener that uses the `json_lines` codec:
+In order for logstash to correctly receive and parse the events, you will need to
+configure and run a listener that uses the `json_lines` codec. For example, to receive
+events over UDP on port 5228:
 
 ```ruby
 input {
@@ -80,14 +83,6 @@ input {
 
 See the [samples](https://github.com/dwbutler/logstash-logger/tree/master/samples) directory for more configuration samples.
 
-## UDP vs TCP
-Should you write to a UDP or TCP listener? It depends on your specific needs, but most applications should use the default (UDP).
-
-* UDP is faster because it's asynchronous (fire-and-forget). However, this means that log messages could get dropped. This is okay for most applications.
-* TCP verifies that every message has been received via two-way communication . This could slow your app down to a crawl if the TCP listener is under heavy load.
-
-For a more detailed discussion of UDP vs TCP, I recommend reading this article: [UDP vs. TCP](http://gafferongames.com/networking-for-game-programmers/udp-vs-tcp/)
-
 ## SSL
 
 If you are using TCP then there is the option of adding an SSL certificate to the options hash on initialize.
@@ -127,21 +122,82 @@ Verified to work with both Rails 3 and 4.
 
 Add the following to your `config/environments/production.rb`:
 
+### Common Options
+
+```ruby
+# Optional, Rails sets the default to :info
+config.log_level = :debug
+
+# Optional, Rails 4 defaults to true in development and false in production
+config.autoflush_log = true
+```
+
+### UDP
 ```ruby
 # Optional, defaults to '0.0.0.0'
 config.logstash.host = 'localhost'
 
-# Required for TCP/UDP
+# Optional, defaults to :udp.
+config.logstash.type = :udp
+
+# Required, the port to connect to
 config.logstash.port = 5228
+```
 
-# Optional, defaults to :udp. Possible values are :udp, :tcp, or :stdout
-config.logstash.type = :udp
+### TCP
+
+```ruby
+# Optional, defaults to '0.0.0.0'
+config.logstash.host = 'localhost'
+
+# Required, the port to connect to
+config.logstash.port = 5228
+
+# Required
+config.logstash.type = :tcp
 
-# Optional, enables SSL when combined with :tcp type
+# Optional, enables SSL
 config.logstash.ssl_enable = true
+```
 
-# Optional, Rails sets the default to :info
-config.log_level = :debug
+### Unix Socket
+
+```ruby
+# Required
+config.logstash.type = :unix
+
+# Required
+config.logstash.path = '/tmp/sock'
+```
+
+### Redis
+
+```ruby
+# Required
+config.logstash.type = :redis
+
+# Optional, will default to the 'logstash' list
+config.logstash.list = 'logstash'
+
+# All other options are passed in to the Redis client
+# Supported options include host, port, path, password, url
+# Example:
+
+# Optional, Redis will default to localhost
+config.logstash.host = 'localhost'
+
+# Optional, Redis will default to port 6379
+config.logstash.port = 6379
+```
+
+### File
+
+```ruby
+# Required
+config.logstash.type = :file
+
+# Optional, defaults to Rails log path
+config.logstash.path = 'log/production.log'
 ```
 
 By default, every Rails log message will be written to logstash in `LogStash::Event` JSON format.
@@ -164,6 +220,25 @@ Verified to work with:
 
 Ruby 1.8.7 is not supported.
 
+
+## What type of logger should I use?
+
+It depends on your specific needs, but most applications should use the default (UDP). Here are the advantages and
+disadvantages of each type:
+
+* UDP is faster than TCP because it's asynchronous (fire-and-forget). However, this means that log messages could get dropped.
+  This is okay for many applications.
+* TCP verifies that every message has been received via two-way communication. It also supports SSL for secure transmission
+  of log messages over a network. This could slow your app down to a crawl if the TCP listener is under heavy load.
+* A file is simple to use, but you will have to worry about log rotation and running out of disk space.
+* Writing to a Unix socket is faster than writing to a TCP or UDP port, but only works locally.
+* Writing to Redis is good for distributed setups that generate tons of logs. However, you will have another moving part and
+  have to worry about Redis running out of memory.
+* Writing to stdout is only recommended for debugging purposes.
+
+For a more detailed discussion of UDP vs TCP, I recommend reading this article:
+[UDP vs. TCP](http://gafferongames.com/networking-for-game-programmers/udp-vs-tcp/)
+
 ## Breaking changes
 
 ### Version 0.5+

data/lib/logstash-logger/device.rb

@@ -5,9 +5,13 @@ module LogStashLogger
     DEFAULT_TYPE = :udp
 
     autoload :Base, 'logstash-logger/device/base'
+    autoload :Connectable, 'logstash-logger/device/connectable'
     autoload :Socket, 'logstash-logger/device/socket'
     autoload :UDP, 'logstash-logger/device/udp'
     autoload :TCP, 'logstash-logger/device/tcp'
+    autoload :Unix, 'logstash-logger/device/unix'
+    autoload :Redis, 'logstash-logger/device/redis'
+    autoload :File, 'logstash-logger/device/file'
     autoload :Stdout, 'logstash-logger/device/stdout'
 
     def self.new(opts)
@@ -20,6 +24,9 @@ module LogStashLogger
       case type
         when :udp then UDP
         when :tcp then TCP
+        when :unix then Unix
+        when :file then File
+        when :redis then Redis
         when :stdout then Stdout
         else fail ArgumentError, 'Invalid type'
       end

data/lib/logstash-logger/device/base.rb

@@ -2,8 +2,10 @@ module LogStashLogger
   module Device
     class Base
       attr_reader :io
+      attr_accessor :sync
 
       def initialize(opts={})
+        @sync = opts[:sync]
       end
 
       def to_io

data/lib/logstash-logger/device/connectable.rb

@@ -0,0 +1,50 @@
+module LogStashLogger
+  module Device
+    class Connectable < Base
+      def write(message)
+        with_connection do
+          super
+        end
+      end
+
+      def flush
+        return unless connected?
+        with_connection do
+          super
+        end
+      end
+
+      def to_io
+        with_connection do
+          @io
+        end
+      end
+
+      def connected?
+        !!@io
+      end
+
+      protected
+
+      # Implemented by subclasses
+      def connect
+        fail NotImplementedError
+      end
+
+      def reconnect
+        @io = nil
+        connect
+      end
+
+      # Ensure the block is executed with a valid connection
+      def with_connection(&block)
+        connect unless @io
+        yield
+      rescue => e
+        warn "#{self.class} - #{e.class} - #{e.message}"
+        close
+        @io = nil
+      end
+    end
+  end
+end

data/lib/logstash-logger/device/file.rb

@@ -0,0 +1,23 @@
+require 'fileutils'
+
+module LogStashLogger
+  module Device
+    class File < Base
+      def initialize(opts)
+        super
+        @path = opts[:path] || fail(ArgumentError, "Path is required")
+        open
+      end
+
+      def open
+        unless ::File.exist? ::File.dirname @path
+          ::FileUtils.mkdir_p ::File.dirname @path
+        end
+
+        @io = ::File.open @path, ::File::WRONLY | ::File::APPEND | ::File::CREAT
+        @io.binmode
+        @io.sync = self.sync
+      end
+    end
+  end
+end

data/lib/logstash-logger/device/redis.rb

@@ -0,0 +1,70 @@
+require 'redis'
+require 'stud/buffer'
+
+module LogStashLogger
+  module Device
+    class Redis < Connectable
+      include Stud::Buffer
+
+      DEFAULT_LIST = 'logstash'
+
+      attr_accessor :list
+
+      def initialize(opts)
+        super
+        @list = opts.delete(:list) || DEFAULT_LIST
+        @redis_options = opts
+
+        @batch_events = opts.fetch(:batch_events, 50)
+        @batch_timeout = opts.fetch(:batch_timeout, 5)
+
+        buffer_initialize max_items: @batch_events, max_interval: @batch_timeout
+      end
+
+      def connect
+        @io = ::Redis.new(@redis_options)
+      end
+
+      def reconnect
+        @io.client.reconnect
+      end
+
+      def with_connection
+        connect unless @io
+        yield
+      rescue ::Redis::InheritedError
+        reconnect
+        retry
+      rescue => e
+        warn "#{self.class} - #{e.class} - #{e.message}"
+        @io = nil
+      end
+
+      def write(message)
+        buffer_receive message, @list
+        buffer_flush(force: true) if @sync
+      end
+
+      def close
+        buffer_flush(final: true)
+        @io && @io.quit
+      rescue => e
+        warn "#{self.class} - #{e.class} - #{e.message}"
+      ensure
+        @io = nil
+      end
+
+      def flush(*args)
+        if args.empty?
+          buffer_flush
+        else
+          messages, list = *args
+          with_connection do
+            @io.rpush(list, messages)
+          end
+        end
+      end
+
+    end
+  end
+end

data/lib/logstash-logger/device/socket.rb

@@ -2,55 +2,16 @@ require 'socket'
 
 module LogStashLogger
   module Device
-    class Socket < Base
+    class Socket < Connectable
       DEFAULT_HOST = '0.0.0.0'
 
       attr_reader :host, :port
 
       def initialize(opts)
+        super
         @port = opts[:port] || fail(ArgumentError, "Port is required")
         @host = opts[:host] || DEFAULT_HOST
       end
-
-      def write(message)
-        with_connection do
-          super
-        end
-      end
-
-      def flush
-        return unless connected?
-        with_connection do
-          super
-        end
-      end
-
-      def to_io
-        with_connection do
-          @io
-        end
-      end
-
-      def connected?
-        !!@io
-      end
-
-      protected
-
-      # Implemented by TCP and UDP devices
-      def connect
-        fail NotImplementedError
-      end
-
-      # Ensure the block is executed with a valid connection
-      def with_connection(&block)
-        connect unless @io
-        yield
-      rescue => e
-        warn "#{self.class} - #{e.class} - #{e.message}"
-        close
-        @io = nil
-      end
     end
   end
 end

data/lib/logstash-logger/device/stdout.rb

@@ -2,7 +2,9 @@ module LogStashLogger
   module Device
     class Stdout < Base
       def initialize(opts={})
+        super
         @io = $stdout
+        @io.sync = sync unless sync.nil?
       end
 
       def close

data/lib/logstash-logger/device/tcp.rb

@@ -29,7 +29,9 @@ module LogStashLogger
       end
 
       def non_ssl_connect
-        @io = TCPSocket.new(@host, @port)
+        @io = TCPSocket.new(@host, @port).tap do |socket|
+          socket.sync = sync unless sync.nil?
+        end
       end
 
       def ssl_connect

data/lib/logstash-logger/device/udp.rb

@@ -4,6 +4,7 @@ module LogStashLogger
       def connect
         @io = UDPSocket.new.tap do |socket|
           socket.connect(@host, @port)
+          socket.sync = sync unless sync.nil?
         end
       end
     end

data/lib/logstash-logger/device/unix.rb

@@ -0,0 +1,18 @@
+require 'socket'
+
+module LogStashLogger
+  module Device
+    class Unix < Connectable
+      def initialize(opts={})
+        super
+        @path = opts[:path] || fail(ArgumentError, "Path is required")
+      end
+
+      def connect
+        @io = ::UNIXSocket.new(@path).tap do |socket|
+          socket.sync = sync unless sync.nil?
+        end
+      end
+    end
+  end
+end

data/lib/logstash-logger/railtie.rb

@@ -6,6 +6,14 @@ module LogStashLogger
 
     logger_options = app.config.logstash
 
+    if logger_options.type == :file
+      logger_options.path ||= app.config.paths["log"].first
+    end
+
+    if app.config.respond_to?(:autoflush_log)
+      logger_options.sync = app.config.autoflush_log
+    end
+
     logger = LogStashLogger.new(logger_options)
 
     logger.level = ::Logger.const_get(app.config.log_level.to_s.upcase)

data/lib/logstash-logger/version.rb

@@ -1,3 +1,3 @@
 module LogStashLogger
-  VERSION = "0.5.0"
+  VERSION = "0.6.0"
 end

data/logstash-logger.gemspec

@@ -19,8 +19,10 @@ Gem::Specification.new do |gem|
   gem.require_paths = ["lib"]
   
   gem.add_runtime_dependency 'logstash-event', '~> 1.2'
+  gem.add_runtime_dependency 'stud'
 
   gem.add_development_dependency 'rails'
+  gem.add_development_dependency 'redis'
   gem.add_development_dependency 'rspec', '>= 3'
   gem.add_development_dependency 'rake'
   gem.add_development_dependency 'pry'

data/samples/file.conf

@@ -0,0 +1,11 @@
+input {
+  file {
+    path => "/usr/local/var/logstash.log"
+    format => json
+  }
+}
+output {
+  stdout {
+    codec => json_lines
+  }
+}

data/samples/redis.conf

@@ -0,0 +1,12 @@
+input {
+  redis {
+    data_type => "list"
+    codec => json_lines
+    key => "logstash"
+  }
+}
+output {
+  stdout {
+    codec => json_lines
+  }
+}

data/samples/unix.conf

@@ -0,0 +1,11 @@
+input {
+  unix {
+    path => "/tmp/logstash"
+    codec => json_lines
+  }
+}
+output {
+  stdout {
+    codec => json_lines
+  }
+}

data/spec/device/file_spec.rb

@@ -0,0 +1,15 @@
+require 'logstash-logger'
+
+describe LogStashLogger::Device::File do
+  include_context 'device'
+
+  it "writes to a file" do
+    expect(file_device.to_io).to be_a ::File
+  end
+
+  context "when path is not specified" do
+    it "raises an exception" do
+      expect { described_class.new }.to raise_error
+    end
+  end
+end

data/spec/device/redis_spec.rb

@@ -0,0 +1,22 @@
+require 'logstash-logger'
+require 'redis'
+
+describe LogStashLogger::Device::Redis do
+  include_context 'device'
+
+  let(:redis) { double("Redis") }
+
+  before(:each) do
+    allow(Redis).to receive(:new) { redis }
+    allow(redis).to receive(:connect)
+  end
+
+  it "writes to a Redis list" do
+    expect(redis).to receive(:rpush)
+    redis_device.write "foo"
+  end
+
+  it "defaults the Redis list to 'logstash'" do
+    expect(redis_device.list).to eq('logstash')
+  end
+end

data/spec/device/tcp_spec.rb

@@ -8,6 +8,8 @@ describe LogStashLogger::Device::TCP do
 
   before(:each) do
     allow(TCPSocket).to receive(:new) { tcp_socket }
+    allow(tcp_socket).to receive(:sync=)
+
     allow(OpenSSL::SSL::SSLSocket).to receive(:new) { ssl_socket }
     allow(ssl_socket).to receive(:connect)
   end

data/spec/device/unix_spec.rb

@@ -0,0 +1,22 @@
+require 'logstash-logger'
+
+describe LogStashLogger::Device::Unix do
+  include_context 'device'
+
+  let(:unix_socket) { double("UNIXSocket") }
+
+  before(:each) do
+    allow(::UNIXSocket).to receive(:new) { unix_socket }
+  end
+
+  it "writes to a local unix socket" do
+    expect(unix_socket).to receive(:write)
+    unix_device.write('foo')
+  end
+
+  context "when path is not specified" do
+    it "raises an exception" do
+      expect { described_class.new }.to raise_error
+    end
+  end
+end

data/spec/spec_helper.rb

@@ -38,4 +38,10 @@ RSpec.shared_context 'device' do
   let(:udp_device) { LogStashLogger::Device.new(type: :udp, port: port) }
   let(:tcp_device) { LogStashLogger::Device.new(type: :tcp, port: port) }
   let(:ssl_tcp_device) { LogStashLogger::Device.new(type: :tcp, port: port, ssl_enable: true) }
+  let(:unix_device) { LogStashLogger::Device.new(type: :unix, path: '/tmp/logstash') }
+
+  let(:file) { Tempfile.new('test') }
+  let(:file_device) { LogStashLogger::Device.new(type: :file, path: file.path)}
+
+  let(:redis_device) { LogStashLogger::Device.new(type: :redis, sync: true) }
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-logger
 version: !ruby/object:Gem::Version
-  version: 0.5.0
+  version: 0.6.0
 platform: ruby
 authors:
 - David Butler
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-06-12 00:00:00.000000000 Z
+date: 2014-06-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: logstash-event
@@ -24,6 +24,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1.2'
+- !ruby/object:Gem::Dependency
+  name: stud
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rails
   requirement: !ruby/object:Gem::Requirement
@@ -38,6 +52,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
@@ -130,10 +158,14 @@ files:
 - lib/logstash-logger.rb
 - lib/logstash-logger/device.rb
 - lib/logstash-logger/device/base.rb
+- lib/logstash-logger/device/connectable.rb
+- lib/logstash-logger/device/file.rb
+- lib/logstash-logger/device/redis.rb
 - lib/logstash-logger/device/socket.rb
 - lib/logstash-logger/device/stdout.rb
 - lib/logstash-logger/device/tcp.rb
 - lib/logstash-logger/device/udp.rb
+- lib/logstash-logger/device/unix.rb
 - lib/logstash-logger/formatter.rb
 - lib/logstash-logger/logger.rb
 - lib/logstash-logger/railtie.rb
@@ -142,13 +174,19 @@ files:
 - logstash-logger.gemspec
 - samples/example.crt
 - samples/example.key
+- samples/file.conf
+- samples/redis.conf
 - samples/ssl.conf
 - samples/tcp.conf
 - samples/udp.conf
+- samples/unix.conf
+- spec/device/file_spec.rb
+- spec/device/redis_spec.rb
 - spec/device/socket_spec.rb
 - spec/device/stdout_spec.rb
 - spec/device/tcp_spec.rb
 - spec/device/udp_spec.rb
+- spec/device/unix_spec.rb
 - spec/device_spec.rb
 - spec/logger_spec.rb
 - spec/rails_spec.rb
@@ -179,10 +217,13 @@ signing_key:
 specification_version: 4
 summary: LogStash Logger for ruby
 test_files:
+- spec/device/file_spec.rb
+- spec/device/redis_spec.rb
 - spec/device/socket_spec.rb
 - spec/device/stdout_spec.rb
 - spec/device/tcp_spec.rb
 - spec/device/udp_spec.rb
+- spec/device/unix_spec.rb
 - spec/device_spec.rb
 - spec/logger_spec.rb
 - spec/rails_spec.rb