lines 0.2.0 → 0.9.1

checksums.yaml

@@ -1,15 +1,7 @@
 ---
-!binary "U0hBMQ==":
-  metadata.gz: !binary |-
-    ZGM0YmFkODI1ZDgwOTA1NTQ1Y2JmZjM5ZDE2ZjIxMzAxM2JhNWZhZg==
-  data.tar.gz: !binary |-
-    YzY3M2FhMGEyNmExZGEyODZiMDIyMDgzOWQ4YTQ4NThiMzM2NDI3NQ==
-!binary "U0hBNTEy":
-  metadata.gz: !binary |-
-    NTI3YzJhY2M5YjMzZDE2NWI3NjdjYmE0MDZjOWU0MDg1MjY0ZTY2NmYzYThh
-    ZjkxNTQ4ZjZjMzQ0OGRiMTZmNTc5ZDhmYjI3ODIwMDg4MDg0NTFmYWE4OWU3
-    MmUyYzgxOTU0MjQ0N2NhNDFmNThkZmU3Y2NjOWMyZWUxMWIwMmU=
-  data.tar.gz: !binary |-
-    Njc3MjM5ZjJmMmY0MWE4OWM1ZmFlYmUyNzdlMmQ0OWFmZTVmZTllZjc0YWRk
-    ZDczMjM4YmEwYTMwYTU1OTU5Y2E1OWIwNjAzMGNiYTc1NTdiNGJhNWUzZjE5
-    ZjhiNDlhYWU1ZGM4MTBmZTQ3YmY4ZmQyNjMxOWU5NDMyZjZiMWU=
+SHA1:
+  metadata.gz: 938854f5278c30f9d2cceb17de859d5293f1eba3
+  data.tar.gz: 76e588d5c75ea0df0cbd138e8c6dce199b27ea85
+SHA512:
+  metadata.gz: a7fcb7a10b27c6900c0bda6365081cefe0b968c5551089f0741490101b4ec88002694f08c145700cabea3433a28e427d08dd3bd54c268d844ebffd0b6ded74a7
+  data.tar.gz: 113168e5ba0ece27faaa8eba50babd78487b74804d27e4a02d1893cae3574a6bdf90cad8bab2a18953bedddbe773b0f8f0f3395bb18c2d680ed0d68b56fd4cfe

data/.gitignore

@@ -1 +1,2 @@
 Gemfile.lock
+*.gem

data/.travis.yml

@@ -3,5 +3,5 @@ language: ruby
 rvm:
   - 1.9.3
   - 2.0.0
+  - 2.1.0
   - jruby-19mode
-  - rbx-19mode

data/CHANGELOG.md

@@ -1,4 +1,20 @@
 
+0.9.1 / 2014-11-12
+==================
+
+Bump after broken release process.
+
+0.9.0 / 2014-11-12
+==================
+
+Moved the logging part to a separate library called u-log.
+http://rubygems.org/gems/u-log
+
+ * REMOVED: All logging parts
+ * NEW: max_bytesize directive to limit line length
+ * FIX: BasicObject serialization for ruby 2.1+
+ * Tons of cleanup
+
 0.2.0 / 2013-07-15 
 ==================
 

data/Gemfile

@@ -1,10 +1,3 @@
 source 'http://rubygems.org'
 gemspec
-group :development do
-  gem 'rdoc'
-end
-group :test do
-  gem 'benchmark-ips'
-  gem 'rake'
-  gem 'rspec'
-end
+gem 'json'

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2013 Jonas Pfenniger
+Copyright (c) 2013 zimbatm
 
 MIT License
 

data/README.md

@@ -1,90 +1,47 @@
 Lines - structured logs for humans
 ==================================
 [![Build
-Status](https://travis-ci.org/zimbatm/lines-ruby.png)](https://travis-ci.org/zimbatm/lines-ruby)
+Status](https://travis-ci.org/zimbatm/lines-ruby.svg?branch=master)](https://travis-ci.org/zimbatm/lines-ruby)
 
-An oppinionated logging library that implement the
+A ruby implementation of the
 [lines](https://github.com/zimbatm/lines) format.
 
-* Log everything in development AND production.
-* Logs should be easy to read, grep and parse.
-* Logging something should never fail.
-* Let the system handle the storage. Write to syslog or STDERR.
-* No log levels necessary. Just log whatever you want.
-
 STATUS: WORK IN PROGRESS
 ========================
 
-Doc is still scarce so it's quite hard to get started. I think reading the
-lib/lines.rb should give a good idea of the capabilities.
-
-Lines.id is a unique ID generator that seems quite handy but I'm not sure if
-it should be part of the lib or not.
-
-It would be nice to expose a method that resolves a context into a hash. It's
-useful to share the context with other tools like an error reporter. Btw,
-Sentry/Raven is great.
-
-There is a parser in the lib but no credible direct consumption path.
-
-Quick intro
------------
+Example
+-------
 
 ```ruby
 require 'lines'
 
-# Setups the outputs. IO and Syslog are supported.
-Lines.use($stdout, Syslog)
+Lines.dump(foo: 3) #=> "foo=3"
 
-# All lines will be prefixed by the global context
-Lines.global['at'] = proc{ Time.now }
+Lines.load("foo=3") #=> {"foo"=>3}
+```
 
-# First example
-Lines.log(foo: 'bar') # logs: at=2013-07-14T14:19:28Z foo=bar
+Uses
+----
 
-# If not a hash, the argument is transformed. A second argument is accepted as
-# a hash
-Lines.log("Hey", count: 3) # logs: at=2013-07-14T14:19:28Z msg=Hey count=3
+CLI pipes format
 
-# You can also keep a context
-class MyClass < ActiveRecord::Base
-  attr_reader :lines
-  def initialize
-    @lines = Lines.context(my_class_id: self.id)
-  end
+Structued logging
 
-  def do_something
-    lines.log("Something happened")
-    # logs: at=2013-07-14T14:19:28Z msg='Something happeend' my_class_id: 2324
-  end
-end
-```
 
-Features
---------
+Generator TODO
+--------------
 
-* Simple to use
-* Thread safe (if the IO#write is)
-* Designed to not raise exceptions (unless it's an IO issue)
-* Lines.logger is a backward-compatible Logger in case you want to retrofit
-* require "lines/active_record" for sane ActiveRecord logs
-* "lines/rack_logger" is a logging middleware for Rack
-* Lines.load and Lines.dump to parse and generate 'lines'
+Add a max_length option
 
-There's also a fork of lograge that you can use with Rails. See
-https://github.com/zimbatm/lograge/tree/lines-output
+Make sure the output is encoded as a UTF-8 string
 
-Known issues
-------------
+Parser TODO
+-----------
 
-Syslog seems to truncate lines longer than 2056 chars and Lines makes if very
-easy to put too much data.
+Implement the max_nesting option
 
-Lines logging speed is reasonable but it could be faster. It writes at around
-5000 lines per second to Syslog on my machine.
+Different parsing modes. Strict and non-strict. Type templates.
+
+Multi-line parsing.
 
-Inspired by
------------
 
- * Scrolls : https://github.com/asenchi/scrolls
- * Lograge : https://github.com/roidrage/lograge

data/examples/cli.rb

@@ -0,0 +1,17 @@
+#!/usr/bin/env ruby
+# Why use -- prefixes in command-line programs ?
+#
+# Here's how we can use lines for a nicer experience:
+#
+# ./cli.rb foo=333 bar=baz 'xxx=[3 4 #t]'
+#
+# Actually zsh makes it less friendly because [] and {} are interpreted
+#
+# FIXME: cli.rb foo='a b'
+# FIXME: cli.rb --foo=abc
+
+$:.unshift File.expand_path('../../lib', __FILE__)
+require 'lines'
+p ARGV
+args = Lines.load(ARGV.join(' '))
+p args

data/lib/lines.rb

@@ -1,413 +1,129 @@
-require 'date'
-require 'time'
-
-# Lines is an opinionated structured log format and a library.
-#
-# Log everything in development AND production.
-# Logs should be easy to read, grep and parse.
-# Logging something should never fail.
-# Let the system handle the storage. Write to syslog or STDERR.
-# No log levels necessary. Just log whatever you want.
-#
-# Example:
-#
-#     log("Oops !", foo: {}, g: [])
-#     #outputs:
-#     # at=2013-03-07T09:21:39+00:00 pid=3242 app=some-process msg="Oops !" foo={} g=[]
-#
-# Usage:
-#
-#     Lines.use(Syslog, $stderr)
-#     Lines.log(foo: 3, msg: "This")
-#
-#     ctx = Lines.context(encoding_id: Log.id)
-#     ctx.log({})
-#
-#     Lines.context(foo: 'bar') do |l|
-#       l.log(items_count: 3)
-#     end
-module Lines
-  class << self
-    attr_reader :global
-    attr_writer :loader, :dumper
-
-    # Parsing object. Responds to #load(string)
-    def loader
-      @loader ||= (
-        require 'lines/loader'
-        Loader
-      )
-    end
-
-    # Serializing object. Responds to #dump(hash)
-    def dumper; @dumper ||= Dumper.new end
-
-    # Returns a backward-compatibile Logger
-    def logger
-      @logger ||= (
-        require 'lines/logger'
-        Logger.new(self)
-      )
-    end
-
-    # Used to configure lines.
-    #
-    # outputs - allows any kind of IO or Syslog
-    #
-    # Usage:
-    #
-    #     Lines.use(Syslog, $stderr, at: proc{ Time.now })
-    def use(*outputs)
-      if outputs.last.kind_of?(Hash)
-        @global = outputs.pop
-      else
-        @global = {}
-      end
-      @outputters = outputs.flatten.map{|o| to_outputter o}
-    end
-
-    # The main function. Used to record objects in the logs as lines.
-    #
-    # obj - a ruby hash. coerced to +{"msg"=>obj}+ otherwise
-    # args - complementary values to put in the line
-    def log(obj, args={})
-      obj = prepare_obj(obj, args)
-      @outputters.each{|out| out.output(dumper, obj) }
-      nil
-    end
-
-    # Add data to the logs
-    #
-    # data - a ruby hash
-    #
-    # return a Context instance
-    def context(data={})
-      new_context = Context.new ensure_hash!(data)
-      yield new_context if block_given?
-      new_context
-    end
-
-    def ensure_hash!(obj) # :nodoc:
-      return {} unless obj
-      return obj if obj.kind_of?(Hash)
-      return obj.to_h if obj.respond_to?(:to_h)
-      {msg: obj}
-    end
-
-    # Parses a lines-formatted string
-    def load(string)
-      loader.load(string)
-    end
-
-    # Generates a lines-formatted string from the given object
-    def dump(obj)
-      dumper.dump ensure_hash!(obj)
-    end
-
-    protected
-
-    def prepare_obj(obj, args={})
-      if obj.kind_of?(Exception)
-        ex = obj
-        obj = {ex: ex.class, msg: ex.to_s}
-        if ex.respond_to?(:backtrace) && ex.backtrace
-          obj[:backtrace] = ex.backtrace
-        end
-      else
-        obj = ensure_hash!(obj)
-      end
-
-      args = ensure_hash!(args)
-
-      g = global.inject({}) do |h, (k,v)|
-        h[k] = (v.respond_to?(:call) ? v.call : v) rescue $!
-        h
-      end
-
-      g.merge(obj.merge(args))
-    end
-
-    def to_outputter(out)
-      return out if out.respond_to?(:output)
-      return StreamOutputter.new(out) if out.respond_to?(:write)
-      return SyslogOutputter.new if out == ::Syslog
-      raise ArgumentError, "unknown outputter #{out.inspect}"
-    end
+# -*- encoding : utf-8 -*-
+
+require 'lines/parser'
+require 'lines/generator'
+require 'lines/version'
+
+# Lines is an opinionated structured log format
+module Lines; extend self
+  # The global default options for the Lines.parse and Lines.load method:
+  #   symbolize_names: false
+  attr_reader :parse_default_options
+  @parse_default_options = {
+    symbolize_names: false,
+  }
+
+  # The global default options for the Lines.dump method:
+  #   max_nesting: 4
+  #   max_size: 2048,
+  attr_reader :generate_default_options
+  @generate_default_options = {
+    max_nesting: 4,
+    max_bytesize: 2048,
+  }
+
+  attr_accessor :parser
+  @parser = Parser
+
+  attr_accessor :generator
+  @generator = Generator
+
+  # Parse the Lines string _source_ into a Ruby data structure and return it.
+  #
+  # _options_ can have the following keys:
+  # * *symbolize_names*: If set to true, returns symbols for the names
+  #   (keys) in a Lines object. Otherwise strings are returned. Strings are
+  #   the default.
+  def parse(source, options = {})
+    opts = Lines.parse_default_options.merge(options.to_hash)
+    @parser.parse(source.to_str, opts)
   end
 
-  # Wrapper object that holds a given context. Emitted by Lines.context
-  class Context
-    attr_reader :data
-
-    def initialize(data)
-      @data = data
-    end
-
-    # Works like the Lines.log method.
-    def log(obj, args={})
-      Lines.log obj, Lines.ensure_hash!(args).merge(data)
-    end
+  # Generate a Lines string from the Ruby data structure _obj_ and return
+  # it.
+  #
+  # _options_ can have the following keys:
+  # * *max_nesting*: The maximum depth of nesting allowed in the data
+  #   structures from which Lines is to be generated. It defaults to 4.
+  # * *max_bytesize*: The maximum number of bytes for a line to be constructed
+  #   on. It defaults to 2048.
+  def generate(obj, options = {})
+    opts = Lines.generate_default_options.merge(options.to_hash)
+    @generator.generate(obj.to_hash, opts)
   end
 
-  # Handles output to any kind of IO
-  class StreamOutputter
-    NL = "\n".freeze
-
-    # stream must accept a #write(str) message
-    def initialize(stream = $stderr)
-      @stream = stream
-      # Is this needed ?
-      @stream.sync = true if @stream.respond_to?(:sync)
-    end
-
-    def output(dumper, obj)
-      str = dumper.dump(obj) + NL
-      @stream.write str
-    end
+  
+  # Load a ruby data structure from a Lines _source_ and return it. A source can
+  # either be a string-like object, an IO-like object, or an object responding
+  # to the read method. If _proc_ was given, it will be called with any nested
+  # Ruby object as an argument recursively in depth first order. To modify the
+  # default options pass in the optional _options_ argument as well.
+  #
+  # The default options for the parser can be changed via the
+  # parse_default_options method.
+  #
+  # This method is part of the implementation of the load/dump interface of
+  # Marshal and YAML.
+  def load(source, proc = nil, options = {})
+    if source.respond_to? :to_str
+      source = source.to_str
+    elsif source.respond_to? :to_io
+      source = source.to_io.read
+    elsif source.respond_to?(:read)
+      source = source.read
+    end
+    result = parse(source, options)
+    recurse_proc(result, &proc) if proc
+    result
   end
 
-  require 'syslog'
-  # Handles output to syslog
-  class SyslogOutputter
-    PRI2SYSLOG = {
-      'debug'    => ::Syslog::LOG_DEBUG,
-      'info'     => ::Syslog::LOG_INFO,
-      'warn'     => ::Syslog::LOG_WARNING,
-      'warning'  => ::Syslog::LOG_WARNING,
-      'err'      => ::Syslog::LOG_ERR,
-      'error'    => ::Syslog::LOG_ERR,
-      'crit'     => ::Syslog::LOG_CRIT,
-      'critical' => ::Syslog::LOG_CRIT,
-    }.freeze
-
-    def initialize(syslog = ::Syslog)
-      @syslog = syslog
-    end
-
-    def output(dumper, obj)
-      prepare_syslog obj[:app]
-
-      obj = obj.dup
-      obj.delete(:pid) # It's going to be part of the message
-      obj.delete(:at)  # Also part of the message
-      obj.delete(:app) # And again
-
-      level = extract_pri(obj)
-
-      @syslog.log(level, "%s", dumper.dump(obj))
-    end
-
-    protected
-
-    def prepare_syslog(app_name)
-      return if @syslog.opened?
-      app_name ||= File.basename($0)
-      @syslog.open(app_name,
-                  ::Syslog::LOG_PID | ::Syslog::LOG_CONS | ::Syslog::LOG_NDELAY,
-                  ::Syslog::LOG_USER)
-    end
-
-    def extract_pri(h)
-      pri = h.delete(:pri).to_s.downcase
-      PRI2SYSLOG[pri] || ::Syslog::LOG_INFO
+  # Recursively calls passed _Proc_ if the parsed data structure is an _Array_ or _Hash_
+  def recurse_proc(result, &proc)
+    case result
+    when Array
+      result.each { |x| recurse_proc x, &proc }
+      proc.call result
+    when Hash
+      result.each { |x, y| recurse_proc x, &proc; recurse_proc y, &proc }
+      proc.call result
+    else
+      proc.call result
     end
   end
+  protected :recurse_proc
 
-  # Some opinions here as well on the format:
-  #
-  # We really want to never fail at dumping because you know, they're logs.
-  # It's better to get a slightly less readable log that no logs at all.
-  #
-  # We're trying to be helpful for humans. It means that if possible we want
-  # to make things shorter and more readable. It also means that ideally
-  # we would like the parsing to be isomorphic but approximations are alright.
-  # For example a symbol might become a string.
-  #
-  # Basically, values are either composite (dictionaries and arrays), quoted
-  # strings or litterals. Litterals are strings that can be parsed to
-  # something else depending if the language supports it or not.
-  # Litterals never contain white-spaces or other weird (very precise !) characters.
+  # Dumps _obj_ as a Lines string, i.e. calls generate on the object and returns
+  # the result.
   #
-  # the true litteral is written as "#t"
-  # the false litteral is written as "#f"
-  # the nil / null litteral is written as "nil"
+  # If anIO (an IO-like object or an object that responds to the write method)
+  # was given, the resulting JSON is written to it.
   #
-  # dictionary keys are always strings or litterals.
+  # If the number of nested arrays or objects exceeds _limit_, the object or
+  # array is replaced with {...} or [...] respectively.
+  # This argument is similar (but not exactly the same!) to the _limit_
+  # argument in Marshal.dump.
   #
-  # Pleaaase, keep units with numbers. And we provide a way for this:
-  # a tuple of (number, litteral) can be concatenated. Eg: (3, 'ms') => 3ms
-  # alternatively if your language supports a time range it could be serialized
-  # to the same value (and parsed back as well).
+  # The default options for the generator can be changed via the
+  # generate_default_options method.
   #
-  # if we don't know how to serialize something we provide a language-specific
-  # string of it and encode is at such.
-  #
-  # The output ought to use the UTF-8 encoding.
-  #
-  # This dumper has been inspired by the OkJSON gem (both formats look alike
-  # after all).
-  class Dumper
-    SPACE = ' '
-    LIT_TRUE = '#t'
-    LIT_FALSE = '#f'
-    LIT_NIL = 'nil'
-    OPEN_BRACE = '{'
-    SHUT_BRACE = '}'
-    OPEN_BRACKET = '['
-    SHUT_BRACKET = ']'
-    SINGLE_QUOTE = "'"
-    DOUBLE_QUOTE = '"'
-
-    constants.each(&:freeze)
-
-    def dump(obj) #=> String
-      objenc_internal(obj)
-    end
-
-    # Used to introduce new ruby litterals.
-    #
-    # Usage:
-    #
-    #     Point = Struct.new(:x, :y)
-    #     Lines.dumper.map(Point) do |p|
-    #       "#{p.x}x#{p.y}"
-    #     end
-    #
-    #     Lines.log msg: Point.new(3, 5)
-    #     # logs: msg=3x5
-    #
-    def map(klass, &rule)
-      @mapping[klass] = rule
-    end
-
-    # After a certain depth, arrays are replaced with [...] and objects with
-    # {...}. Default is 4.
-    attr_accessor :max_depth
-
-    protected
-
-    attr_reader :mapping
-
-    def initialize
-      @mapping = {}
-      @max_depth = 4
-    end
-
-    def objenc_internal(x, depth=0)
-      depth += 1
-      if depth > max_depth
-        '...'
-      else
-        x.map{|k,v| "#{keyenc(k)}=#{valenc(v, depth)}" }.join(SPACE)
-      end
-    end
-
-    def keyenc(k)
-      case k
-      when String, Symbol then strenc(k)
-      else
-        strenc(k.inspect)
-      end
-    end
-
-    def valenc(x, depth)
-      case x
-      when Hash           then objenc(x, depth)
-      when Array          then arrenc(x, depth)
-      when String, Symbol then strenc(x)
-      when Numeric        then numenc(x)
-      when Time           then timeenc(x)
-      when Date           then dateenc(x)
-      when true           then LIT_TRUE
-      when false          then LIT_FALSE
-      when nil            then LIT_NIL
-      else
-        litenc(x)
-      end
-    end
-
-    def objenc(x, depth)
-      OPEN_BRACE + objenc_internal(x, depth) + SHUT_BRACE
-    end
-
-    def arrenc(a, depth)
-      depth += 1
-      # num + unit. Eg: 3ms
-      if a.size == 2 && a.first.kind_of?(Numeric) && is_literal?(a.last.to_s)
-        "#{numenc(a.first)}:#{strenc(a.last)}"
-      elsif depth > max_depth
-        '[...]'
-      else
-        OPEN_BRACKET + a.map{|x| valenc(x, depth)}.join(' ') + SHUT_BRACKET
-      end
-    end
-
-    def strenc(s)
-      s = s.to_s
-      unless is_literal?(s)
-        s = s.inspect
-        unless s[1..-2].include?(SINGLE_QUOTE)
-          s.gsub!(SINGLE_QUOTE, "\\'")
-          s.gsub!('\"', DOUBLE_QUOTE)
-          s[0] = s[-1] = SINGLE_QUOTE
-        end
-      end
-      s
-    end
-
-    def numenc(n)
-      #case n
-      # when Float
-      #   "%.3f" % n
-      #else
-        n.to_s
-      #end
-    end
-
-    def litenc(x)
-      klass = (x.class.ancestors & mapping.keys).first
-      if klass
-        mapping[klass].call(x)
-      else
-        strenc(x.inspect)
+  # This method is part of the implementation of the load/dump interface of
+  # Marshal and YAML.
+  def dump(obj, anIO = nil, limit = nil)
+    if anIO and limit.nil?
+      anIO = anIO.to_io if anIO.respond_to?(:to_io)
+      unless anIO.respond_to?(:write)
+        limit = anIO
+        anIO = nil
       end
-    rescue
-      klass = (class << x; self; end).ancestors.first
-      strenc("#<#{klass}:0x#{x.__id__.to_s(16)}>")
-    end
-
-    def timeenc(t)
-      t.utc.iso8601
-    end
-
-    def dateenc(d)
-      d.iso8601
     end
-
-    def is_literal?(s)
-      !s.index(/[\s'"=:{}\[\]]/)
-    end
-
-  end
-
-  require 'securerandom'
-  module UniqueIDs
-    # A small utility to generate unique IDs that are as short as possible.
-    #
-    # It's useful to link contextes together
-    #
-    # See http://preshing.com/20110504/hash-collision-probabilities
-    def id(collision_chance=1.0/10e9, over_x_messages=10e3)
-      # Assuming that the distribution is perfectly random
-      # how many bits do we need so that the chance of collision over_x_messages
-      # is lower thant collision_chance ? 
-      number_of_possible_numbers = (over_x_messages ** 2) / (2 * collision_chance)
-      num_bytes = (Math.log2(number_of_possible_numbers) / 8).ceil
-      SecureRandom.urlsafe_base64(num_bytes)
+    opts = {}
+    opts[:max_nesting] = limit if limit
+    result = generate(obj, opts)
+    if anIO
+      anIO.write result
+      anIO
+    else
+      result
     end
   end
-  extend UniqueIDs
 end
-
-# default config
-Lines.use($stderr)