yajl-ffi 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 1b714671d249f07a8fd35a4f13a236a572a57f17
+  data.tar.gz: 8449424c54943747b93f9927e5bacc24969099d9
+SHA512:
+  metadata.gz: d1c3aaeeb4c9d8af71712bfbe420ec93c3ef9a664f59227eb619f1bc561c926b07ca82953317d015240ff594e61a61bf281d54ce8ba1b6a22829b5d453577bcf
+  data.tar.gz: da1ea8c2301ea0020361de9cc56acde0fbe3bf13baeef046123c2d350cfe00a490b1e07f27cf8aa27909aef25c7d5033d3b5315f1a0a5c9e58d18de67bc3ce48

data/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2014 David Graham
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,180 @@
+# Yajl::FFI
+
+Yajl::FFI is a [JSON](http://json.org) parser, based on
+[FFI](https://github.com/ffi/ffi) bindings into the native
+[YAJL](https://github.com/lloyd/yajl) library, that generates
+events for each state change. This allows streaming both the JSON document into
+memory and the parsed object graph out of memory to some other process.
+
+This is similar to an XML SAX parser that generates events during parsing. There
+is no requirement for the document, or the object graph, to be fully buffered in
+memory. Yajl::FFI is best suited for huge JSON documents that won't fit in memory.
+
+## Usage
+
+The simplest way to parse is to read the full JSON document into memory
+and then parse it into a full object graph. This is fine for small documents
+because we have room for both the text and parsed object in memory.
+
+```ruby
+require 'yajl/ffi'
+json = File.read('/tmp/test.json')
+obj = Yajl::FFI::Parser.parse(json)
+```
+
+While it's possible to do this with Yajl::FFI, we should really use the
+standard library's [json]([json](https://github.com/flori/json)
+gem for documents like this. It's faster because it doesn't need to generate
+events and notify observers each time the parser changes state. It parses and
+builds the Ruby object entirely in native code and hands it back to us, fully
+formed.
+
+For larger documents, we can use an IO object to stream it into the parser.
+We still need room for the parsed object, but the document itself is never
+fully read into memory.
+
+```ruby
+require 'yajl/ffi'
+stream = File.open('/tmp/test.json')
+obj = Yajl::FFI::Parser.parse(stream)
+```
+
+However, when streaming small documents from disk, or over the network, the
+[yajl-ruby](https://github.com/brianmario/yajl-ruby) gem will give us the best
+performance.
+
+Huge documents arriving over the network in small chunks to an
+[EventMachine](https://github.com/eventmachine/eventmachine)
+`receive_data` loop is where Yajl::FFI is uniquely suited. Inside an
+`EventMachine::Connection` subclass we might have:
+
+```ruby
+def post_init
+  @parser = Yajl::FFI::Parser.new
+  @parser.start_document { puts "start document" }
+  @parser.end_document   { puts "end document" }
+  @parser.start_object   { puts "start object" }
+  @parser.end_object     { puts "end object" }
+  @parser.start_array    { puts "start array" }
+  @parser.end_array      { puts "end array" }
+  @parser.key            {|k| puts "key: #{k}" }
+  @parser.value          {|v| puts "value: #{v}" }
+end
+
+def receive_data(data)
+  begin
+    @parser << data
+  rescue Yajl::FFI::ParserError => e
+    close_connection
+  end
+end
+```
+
+The parser accepts chunks of the JSON document and parses up to the end of the
+available buffer. Passing in more data resumes the parse from the prior state.
+When an interesting state change happens, the parser notifies all registered
+callback procs of the event.
+
+The event callback is where we can do interesting data filtering and passing
+to other processes. The above example simply prints state changes, but the
+callbacks might look for an array named `rows` and process sets of these row
+objects in small batches. Millions of rows, streaming over the network, can be
+processed in constant memory space this way.
+
+## Dependencies
+
+* [libyajl2](https://github.com/lloyd/yajl)
+* ruby >= 1.9.3
+* jruby >= 1.7
+
+## Library loading
+
+FFI uses the the `dlopen` system call to dynamically load the libyajl library
+into memory at runtime. It searches the usual directories for the library file,
+like `/usr/lib` and `/usr/local/lib`, and raises an error if it's not found.
+If libyajl is installed in an unusual directory, we can tell `dlopen` where to
+look by setting the `LD_LIBRARY_PATH` environment variable.
+
+```sh
+# test normal library load
+$ ruby -r 'yajl/ffi' -e 'puts Yajl::FFI::VERSION'
+
+# if it fails, specify the search path
+$ LD_LIBRARY_PATH=/somewhere/yajl/lib \
+  ruby -r 'yajl/ffi' -e 'puts Yajl::FFI::VERSION'
+```
+
+## Installation
+
+The libyajl library needs to be installed before this gem can bind to it.
+
+### OS X
+
+Use [Homebrew](http://brew.sh) or compile from source below.
+
+```
+$ brew install yajl
+```
+
+### Fedora
+
+Fedora 20 provides libyajl2 in a package. Older versions might need to compile
+the latest yajl version from source.
+
+```
+$ sudo yum install yajl
+```
+
+### Ubuntu
+
+Ubuntu 14.04 provides a libyajl2 package. Older versions might also need to
+compile yajl from source.
+
+```
+$ sudo apt-get install libyajl2
+```
+
+### Source
+
+By default, this compiles and installs to `/usr/local`. Use
+`./configure -p /tmp/somewhere` to install to a different directory.
+Setting `LD_LIBRARY_PATH` will be required in that case.
+
+```
+$ git clone https://github.com/lloyd/yajl
+$ cd yajl
+$ ./configure
+$ make && make install
+```
+
+## Alternatives
+
+* [json](https://github.com/flori/json)
+* [yajl-ruby](https://github.com/brianmario/yajl-ruby)
+* [json-stream](https://github.com/dgraham/json-stream)
+
+This gem provides a benchmark script to test the relative performance of
+several parsers. Here's a sample run.
+
+```
+$ rake benchmark
+                  user     system      total        real
+json          0.130000   0.010000   0.140000 (  0.136225)
+yajl-ruby     0.130000   0.010000   0.140000 (  0.133872)
+yajl-ffi      0.800000   0.010000   0.810000 (  0.812818)
+json-stream   9.500000   0.080000   9.580000 (  9.580571)
+```
+
+Yajl::FFI is about 6x slower than the pure native parsers. JSON::Stream is a
+pure Ruby parser, and it performs accordingly. But it's useful in cases where
+you're unable to use native bindings or when the limiting factor is the
+network, rather than processor speed.
+
+So if you need to parse many small JSON documents, the json and yajl-ruby gems
+are the best options. If you need to stream, and incrementally parse, pieces of a
+large document in constant memory space, yajl-ffi and json-stream are good
+choices.
+
+## License
+
+Yajl::FFI is released under the MIT license. Check the LICENSE file for details.

data/Rakefile

@@ -0,0 +1,22 @@
+require 'rake'
+require 'rake/clean'
+require 'rake/testtask'
+
+load './lib/yajl/ffi/tasks/benchmark.rake'
+
+CLOBBER.include('pkg')
+
+directory 'pkg'
+
+desc 'Build distributable packages'
+task :build => [:pkg] do
+  system 'gem build yajl-ffi.gemspec && mv yajl-ffi-*.gem pkg/'
+end
+
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'test'
+  test.pattern = 'spec/**/*_spec.rb'
+  test.warning = true
+end
+
+task :default => [:clobber, :test, :build]

data/lib/yajl/ffi.rb

@@ -0,0 +1,54 @@
+require 'ffi'
+require 'stringio'
+require 'yajl/ffi/builder'
+require 'yajl/ffi/parser'
+require 'yajl/ffi/version'
+
+module Yajl
+  module FFI
+    extend ::FFI::Library
+
+    ffi_lib 'yajl'
+
+    enum :status, [
+      :ok,
+      :client_canceled,
+      :error
+    ]
+
+    enum :options, [
+      :allow_comments,         0x01,
+      :allow_invalid_utf8,     0x02,
+      :allow_trailing_garbage, 0x04,
+      :allow_multiple_values,  0x08,
+      :allow_partial_values,   0x10
+    ]
+
+    class Callbacks < ::FFI::Struct
+      layout \
+        :on_null,         :pointer,
+        :on_boolean,      :pointer,
+        :on_integer,      :pointer,
+        :on_double,       :pointer,
+        :on_number,       :pointer,
+        :on_string,       :pointer,
+        :on_start_object, :pointer,
+        :on_key,          :pointer,
+        :on_end_object,   :pointer,
+        :on_start_array,  :pointer,
+        :on_end_array,    :pointer
+    end
+
+    typedef :pointer, :handle
+
+    attach_function :alloc,              :yajl_alloc,              [:pointer, :pointer, :pointer],     :handle
+    attach_function :free,               :yajl_free,               [:handle],                          :void
+    attach_function :config,             :yajl_config,             [:handle, :options, :varargs],      :int
+    attach_function :parse,              :yajl_parse,              [:handle, :pointer, :size_t],       :status
+    attach_function :complete_parse,     :yajl_complete_parse,     [:handle],                          :status
+    attach_function :get_error,          :yajl_get_error,          [:handle, :int, :pointer, :size_t], :pointer
+    attach_function :free_error,         :yajl_free_error,         [:handle, :pointer],                :void
+    attach_function :get_bytes_consumed, :yajl_get_bytes_consumed, [:handle],                          :size_t
+    attach_function :status_to_string,   :yajl_status_to_string,   [:status],                          :string
+  end
+end

data/lib/yajl/ffi/builder.rb

@@ -0,0 +1,70 @@
+module Yajl
+  module FFI
+    # A parser listener that builds a full, in memory, object from a JSON
+    # document. This is similar to using the json gem's `JSON.parse` method.
+    #
+    # Examples
+    #
+    #   parser = Yajl::FFI::Parser.new
+    #   builder = Yajl::FFI::Builder.new(parser)
+    #   parser << '{"answer": 42, "question": false}'
+    #   obj = builder.result
+    class Builder
+      METHODS = %w[start_document end_document start_object end_object start_array end_array key value]
+
+      attr_reader :result
+
+      def initialize(parser)
+        METHODS.each do |name|
+          parser.send(name, &method(name))
+        end
+      end
+
+      def start_document
+        @stack = []
+        @keys = []
+        @result = nil
+      end
+
+      def end_document
+        @result = @stack.pop
+      end
+
+      def start_object
+        @stack.push({})
+      end
+
+      def end_object
+        return if @stack.size == 1
+        node = @stack.pop
+
+        case @stack.last
+        when Hash
+          @stack.last[@keys.pop] = node
+        when Array
+          @stack.last << node
+        end
+      end
+      alias :end_array :end_object
+
+      def start_array
+        @stack.push([])
+      end
+
+      def key(key)
+        @keys << key
+      end
+
+      def value(value)
+        case @stack.last
+        when Hash
+          @stack.last[@keys.pop] = value
+        when Array
+          @stack.last << value
+        else
+          @stack << value
+        end
+      end
+    end
+  end
+end

data/lib/yajl/ffi/parser.rb

@@ -0,0 +1,316 @@
+module Yajl
+  module FFI
+    # Raised on any invalid JSON text.
+    ParserError = Class.new(RuntimeError)
+
+    # A streaming JSON parser that generates SAX-like events for state changes.
+    #
+    # Examples
+    #
+    #   parser = Yajl::FFI::Parser.new
+    #   parser.key {|key| puts key }
+    #   parser.value {|value| puts value }
+    #   parser << '{"answer":'
+    #   parser << ' 42}'
+    class Parser
+      BUF_SIZE = 4096
+      CONTINUE_PARSE = 1
+      FLOAT = /[\.eE]/
+
+      # Parses a full JSON document from a String or an IO stream and returns
+      # the parsed object graph. For parsing small JSON documents with small
+      # memory requirements, use the json gem's faster JSON.parse method instead.
+      #
+      # json - The String or IO containing JSON data.
+      #
+      # Examples
+      #
+      #   Yajl::FFI::Parser.parse('{"hello": "world"}')
+      #   # => {"hello": "world"}
+      #
+      # Raises a Yajl::FFI::ParserError if the JSON data is malformed.
+      #
+      # Returns a Hash.
+      def self.parse(json)
+        stream = json.is_a?(String) ? StringIO.new(json) : json
+        parser = Parser.new
+        builder = Builder.new(parser)
+        while (buffer = stream.read(BUF_SIZE)) != nil
+          parser << buffer
+        end
+        parser.finish
+        builder.result
+      ensure
+        stream.close
+      end
+
+      # Create a new parser with an optional initialization block where
+      # we can register event callbacks.
+      #
+      # Examples
+      #
+      #   parser = Yajl::FFI::Parser.new do
+      #     start_document { puts "start document" }
+      #     end_document   { puts "end document" }
+      #     start_object   { puts "start object" }
+      #     end_object     { puts "end object" }
+      #     start_array    { puts "start array" }
+      #     end_array      { puts "end array" }
+      #     key            {|k| puts "key: #{k}" }
+      #     value          {|v| puts "value: #{v}" }
+      #   end
+      def initialize(&block)
+        @listeners = {
+          start_document: [],
+          end_document: [],
+          start_object: [],
+          end_object: [],
+          start_array: [],
+          end_array: [],
+          key: [],
+          value: []
+        }
+
+        # Track parse stack.
+        @depth = 0
+        @started = false
+
+        # Allocate native memory.
+        @callbacks = callbacks
+        @handle = Yajl::FFI.alloc(@callbacks.to_ptr, nil, nil)
+        @handle = ::FFI::AutoPointer.new(@handle, method(:release))
+
+        # Register any observers in the block.
+        instance_eval(&block) if block_given?
+      end
+
+      def start_document(&block)
+        @listeners[:start_document] << block
+      end
+
+      def end_document(&block)
+        @listeners[:end_document] << block
+      end
+
+      def start_object(&block)
+        @listeners[:start_object] << block
+      end
+
+      def end_object(&block)
+        @listeners[:end_object] << block
+      end
+
+      def start_array(&block)
+        @listeners[:start_array] << block
+      end
+
+      def end_array(&block)
+        @listeners[:end_array] << block
+      end
+
+      def key(&block)
+        @listeners[:key] << block
+      end
+
+      def value(&block)
+        @listeners[:value] << block
+      end
+
+      # Pass data into the parser to advance the state machine and
+      # generate callback events. This is well suited for an EventMachine
+      # receive_data loop.
+      #
+      # data - The String of partial JSON data to parse.
+      #
+      # Raises a Yajl::FFI::ParserError if the JSON data is malformed.
+      #
+      # Returns nothing.
+      def <<(data)
+        result = Yajl::FFI.parse(@handle, data, data.bytesize)
+        error(data) if result == :error
+        if @started && @depth == 0
+          result = Yajl::FFI.complete_parse(@handle)
+          error(data) if result == :error
+        end
+      end
+
+      # Drain any remaining buffered characters into the parser to complete
+      # the parsing of the document.
+      #
+      # This is only required when parsing a document containing a single
+      # numeric value, integer or float. The parser has no other way to
+      # detect when it should no longer expect additional characters with
+      # which to complete the parse, so it must be signaled by a call to
+      # this method.
+      #
+      # If you're parsing more typical object or array documents, there's no
+      # need to call `finish` because the parse will complete when the final
+      # closing `]` or `}` character is scanned.
+      #
+      # Raises a Yajl::FFI::ParserError if the JSON data is malformed.
+      #
+      # Returns nothing.
+      def finish
+        result = Yajl::FFI.complete_parse(@handle)
+        error('') if result == :error
+      end
+
+      private
+
+      # Raise a ParserError for the malformed JSON data sent to the parser.
+      #
+      # data - The malformed JSON String that the yajl parser rejected.
+      #
+      # Returns nothing.
+      def error(data)
+        pointer = Yajl::FFI.get_error(@handle, 1, data, data.bytesize)
+        message = pointer.read_string
+        Yajl::FFI.free_error(@handle, pointer)
+        raise ParserError, message
+      end
+
+      # Invoke all registered observer procs for the event type.
+      #
+      # type - The Symbol listener name.
+      # args - The argument list to pass into the observer procs.
+      #
+      # Examples
+      #
+      #    # broadcast events for {"answer": 42}
+      #    notify(:start_object)
+      #    notify(:key, "answer")
+      #    notify(:value, 42)
+      #    notify(:end_object)
+      #
+      # Returns nothing.
+      def notify(type, *args)
+        @started = true
+        @listeners[type].each do |block|
+          block.call(*args)
+        end
+      end
+
+      # Build a native Callbacks struct that broadcasts parser state change
+      # events to registered observers.
+      #
+      # The functions registered in the struct are invoked by the native yajl
+      # parser. They convert the yajl callback data into the expected Ruby
+      # objects and invoke observers registered on the parser with
+      # `start_object`, `key`, `value`, and so on.
+      #
+      # The struct instance returned from this method must be stored in an
+      # instance variable. This prevents the FFI::Function objects from being
+      # garbage collected while the parser is still in use. The native function
+      # bindings need to be collected at the same time as the Parser instance.
+      #
+      # Returns a Yajl::FFI::Callbacks struct.
+      def callbacks
+        callbacks = Yajl::FFI::Callbacks.new
+
+        callbacks[:on_null] = ::FFI::Function.new(:int, [:pointer]) do |ctx|
+          notify(:start_document) if @depth == 0
+          notify(:value, nil)
+          notify(:end_document) if @depth == 0
+          CONTINUE_PARSE
+        end
+
+        callbacks[:on_boolean] = ::FFI::Function.new(:int, [:pointer, :int]) do |ctx, value|
+          notify(:start_document) if @depth == 0
+          notify(:value, value == 1)
+          notify(:end_document) if @depth == 0
+          CONTINUE_PARSE
+        end
+
+        # yajl only calls on_number
+        callbacks[:on_integer] = nil
+        callbacks[:on_double] = nil
+
+        callbacks[:on_number] = ::FFI::Function.new(:int, [:pointer, :string, :size_t]) do |ctx, value, length|
+          notify(:start_document) if @depth == 0
+          value = value.slice(0, length)
+          number = (value =~ FLOAT) ? value.to_f : value.to_i
+          notify(:value, number)
+          notify(:end_document) if @depth == 0
+          CONTINUE_PARSE
+        end
+
+        callbacks[:on_string] = ::FFI::Function.new(:int, [:pointer, :pointer, :size_t]) do |ctx, value, length|
+          notify(:start_document) if @depth == 0
+          notify(:value, extract(value, length))
+          notify(:end_document) if @depth == 0
+          CONTINUE_PARSE
+        end
+
+        callbacks[:on_start_object] = ::FFI::Function.new(:int, [:pointer]) do |ctx|
+          @depth += 1
+          notify(:start_document) if @depth == 1
+          notify(:start_object)
+          CONTINUE_PARSE
+        end
+
+        callbacks[:on_key] = ::FFI::Function.new(:int, [:pointer, :pointer, :size_t]) do |ctx, key, length|
+          notify(:key, extract(key, length))
+          CONTINUE_PARSE
+        end
+
+        callbacks[:on_end_object] = ::FFI::Function.new(:int, [:pointer]) do |ctx|
+          @depth -= 1
+          notify(:end_object)
+          notify(:end_document) if @depth == 0
+          CONTINUE_PARSE
+        end
+
+        callbacks[:on_start_array] = ::FFI::Function.new(:int, [:pointer]) do |ctx|
+          @depth += 1
+          notify(:start_document) if @depth == 1
+          notify(:start_array)
+          CONTINUE_PARSE
+        end
+
+        callbacks[:on_end_array] = ::FFI::Function.new(:int, [:pointer]) do |ctx|
+          @depth -= 1
+          notify(:end_array)
+          notify(:end_document) if @depth == 0
+          CONTINUE_PARSE
+        end
+
+        callbacks
+      end
+
+      # Convert the binary encoded string data passed out of the yajl parser
+      # into a UTF-8 encoded string.
+      #
+      # pointer - The FFI::Pointer containing the ASCII-8BIT encoded String.
+      # length  - The Fixnum number of characters to extract from `pointer`.
+      #
+      # Raises a ParserError if the data contains malformed UTF-8 bytes.
+      #
+      # Returns a String.
+      def extract(pointer, length)
+        string = pointer.read_string(length)
+        string.force_encoding(Encoding::UTF_8)
+        unless string.valid_encoding?
+          raise ParserError, 'Invalid UTF-8 byte sequence'
+        end
+        string
+      end
+
+      # Free the memory held by a yajl parser handle previously allocated
+      # with Yajl::FFI.alloc.
+      #
+      # It's not sufficient to just allow the handle pointer to be freed
+      # normally because it contains pointers that must also be freed. The
+      # native yajl API provides a `yajl_free` function for this purpose.
+      #
+      # This method is invoked by the FFI::AutoPointer, wrapping the yajl
+      # parser handle, when it's garbage collected by Ruby.
+      #
+      # pointer - The FFI::Pointer that references the native yajl parser.
+      #
+      # Returns nothing.
+      def release(pointer)
+        Yajl::FFI.free(pointer)
+      end
+    end
+  end
+end