json-stream 0.1.0

data/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2010 David Graham
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README

@@ -0,0 +1,82 @@
+== Welcome to JSON::Stream
+
+JSON::Stream is a finite state machine based JSON parser that generates events
+for each state change. This allows us to stream both the JSON document into
+memory and the parsed object graph out of memory to some other process.  This
+is much like an XML SAX parser that generates events during parsing.  There is
+no requirement for the document nor the object graph to be fully buffered in
+memory.  This is best suited for huge JSON documents that won't fit in memory.
+For example, streaming and processing large map/reduce views from Apache CouchDB.
+
+== Usage
+
+The simplest way to parse is to read the full JSON document into memory
+and then parse it into a full object graph.  This is fine for small documents
+because we have room for both the document and parsed object in memory.
+
+require 'json/stream'
+json = File.read('/tmp/test.json')
+obj = JSON::Stream::Parser.parse(json)
+
+While it's possible to do this with JSON::Stream, we really want to use the json
+gem for documents like this.  JSON.parse() is much faster than this parser
+because it can rely on having the entire document in memory to analyze.
+
+For larger documents we can use an IO object to stream it into the parser.
+We still need room for the parsed object, but the document itself is never
+fully read into memory.
+
+require 'json/stream'
+stream = File.open('/tmp/test.json')
+obj = JSON::Stream::Parser.parse(stream)
+
+Again, while we can do this with JSON::Stream, if we just need to stream the
+document from disk or the network, we're better off using the yajl-ruby gem.
+
+Huge documents arriving over the network in small chunks to an EventMachine
+receive_data loop is where JSON::Stream is really useful.  Inside our
+EventMachine::Connection subclass we might have:
+
+def post_init
+  @parser = JSON::Stream::Parser.new do
+    start_document { puts "start document" }
+    end_document   { puts "end document" }
+    start_object   { puts "start object" }
+    end_object     { puts "end object" }
+    start_array    { puts "start array" }
+    end_array      { puts "end array" }
+    key            {|k| puts "key: #{k}" }
+    value          {|v| puts "value: #{v}" }
+  end
+end
+
+def receive_data(data)
+  begin
+    @parser << data
+  rescue JSON::Stream::ParserError => e
+    close_connection
+  end
+end
+
+Notice how the parser accepts chunks of the JSON document and parses up
+up to the end of the available buffer.  Passing in more data resumes the
+parse from the prior state.  When an interesting state change happens, the
+parser notifies all registered callback procs of the event.
+
+The event callback is where we can do interesting data filtering and passing
+to other processes.  The above example simply prints state changes, but
+imagine the callbacks looking for an array named "rows" and processing sets
+of these row objects in small batches.  We can process millions of rows streaming
+over the network in constant memory space this way.
+
+== Dependencies
+
+* ruby >= 1.9.1
+
+== Contact
+
+Project contact: David Graham <david.malcom.graham@gmail.com>
+
+== License
+
+JSON::Stream is released under the MIT license.  Check the LICENSE file for details.

data/Rakefile

@@ -0,0 +1,38 @@
+require 'rake'
+require 'rake/clean'
+require 'rake/gempackagetask'
+require 'rake/testtask'
+require File.join(File.expand_path(File.dirname(__FILE__)), 'lib', 'json', 'stream')
+
+spec = Gem::Specification.new do |s| 
+  s.name = "json-stream"
+  s.version = JSON::Stream::VERSION
+  s.date = Time.now.strftime("%Y-%m-%d")
+  s.summary = "A streaming JSON parser that generates SAX-like events."
+  s.description = "A finite state machine based JSON parser that generates events
+for each state change. This allows us to stream both the JSON document into
+memory and the parsed object graph out of memory to some other process.  This
+is much like an XML SAX parser that generates events during parsing.  There is
+no requirement for the document nor the object graph to be fully buffered in
+memory.  This is best suited for huge JSON documents that won't fit in memory.
+For example, streaming and processing large map/reduce views from Apache CouchDB."
+  s.email = "david.malcom.graham@gmail.com"
+  s.homepage = "http://github.com/dgraham/json-stream"
+  s.authors = ["David Graham"]
+  s.files = FileList['LICENSE', 'README', 'Rakefile', "{lib}/**/*"].to_a
+  s.require_path = "lib"
+  s.test_files = FileList["{test}/**/*test.rb"].to_a
+  s.has_rdoc = true
+  s.required_ruby_version = '>= 1.9.1'
+end
+ 
+Rake::GemPackageTask.new(spec) do |pkg|
+  pkg.need_tar = true 
+end 
+
+Rake::TestTask.new(:test) do |test|
+  test.pattern = 'test/**/*_test.rb'
+  test.warning = true
+end
+
+task :default => [:clobber, :test, :gem]

data/lib/json/stream.rb

@@ -0,0 +1,15 @@
+# encoding: UTF-8
+
+$:.unshift File.dirname(__FILE__) unless
+  $:.include?(File.dirname(__FILE__))
+
+require 'stringio'
+require 'stream/buffer'
+require 'stream/builder'
+require 'stream/parser'
+
+module JSON
+  module Stream
+    VERSION = "0.1.0"
+  end
+end

data/lib/json/stream/buffer.rb

@@ -0,0 +1,63 @@
+# encoding: UTF-8
+
+module JSON
+  module Stream
+
+    # A character buffer that expects a UTF-8 encoded stream of bytes.
+    # This handles truncated multi-byte characters properly so we can just
+    # feed it binary data and receive a properly formatted UTF-8 String as
+    # output. See here for UTF-8 parsing details:
+    # http://en.wikipedia.org/wiki/UTF-8
+    # http://tools.ietf.org/html/rfc3629#section-3
+    class Buffer
+      def initialize
+        @state, @buf, @need = :start, [], 0
+      end
+
+      # Fill the buffer with a String of binary UTF-8 encoded bytes. Returns
+      # as much of the data in a UTF-8 String as we have. Truncated multi-byte
+      # characters are saved in the buffer until the next call to this method
+      # where we expect to receive the rest of the multi-byte character.
+      def <<(data)
+        bytes = []
+        data.bytes.each do |b|
+          case @state
+          when :start
+            if b < 128
+              bytes << b
+            elsif b >= 192
+              @state = :multi_byte
+              @buf << b
+              @need = case
+                when b >= 240 then 4
+                when b >= 224 then 3
+                when b >= 192 then 2 end
+            else
+              error('Expected start of multi-byte or single byte char')
+            end
+          when :multi_byte
+            if b > 127 && b < 192
+              @buf << b
+              if @buf.size == @need
+                bytes += @buf.slice!(0, @buf.size)
+                @state = :start
+              end
+            else
+              error('Expected continuation byte')
+            end
+          end
+        end
+        bytes.pack('C*').force_encoding(Encoding::UTF_8).tap do |str|
+          error('Invalid UTF-8 byte sequence') unless str.valid_encoding?
+        end
+      end
+
+      private
+
+      def error(message)
+        raise ParserError, message
+      end
+    end
+
+  end
+end

data/lib/json/stream/builder.rb

@@ -0,0 +1,92 @@
+# encoding: UTF-8
+
+module JSON
+  module Stream
+    # A parser listener that builds a full, in memory, object graph from a
+    # JSON document. Typically, we would use the json gem's JSON.parse() method
+    # when we have the full JSON document because it's much faster than this.
+    # JSON::Stream is typically used when we have a huge JSON document streaming
+    # to us and we don't want to hold the entire parsed object in memory.
+    # Regardless, this is a good example of how to write parser callbacks.
+    #
+    # parser = JSON::Stream::Parser.new
+    # builder = JSON::Stream::Builder.new(parser)
+    # parser << json
+    # obj = builder.result
+    class Builder
+      METHODS = %w[start_document end_document start_object end_object start_array end_array key value]
+
+      attr_reader :result
+
+      def initialize(parser)
+        METHODS.each do |name|
+          parser.send(name, &method(name))
+        end
+      end
+
+      def start_document
+        @stack, @result = [], nil
+      end
+
+      def end_document
+        @result = @stack.pop.obj
+      end
+
+      def start_object
+        @stack.push(ObjectNode.new)
+      end
+
+      def end_object
+        unless @stack.size == 1
+          node = @stack.pop
+          @stack[-1] << node.obj
+        end
+      end
+      alias :end_array :end_object
+
+      def start_array
+        @stack.push(ArrayNode.new)
+      end
+
+      def key(key)
+        @stack[-1] << key
+      end
+
+      def value(value)
+        @stack[-1] << value
+      end
+    end
+
+    class ArrayNode
+      attr_reader :obj
+
+      def initialize
+        @obj = []
+      end
+
+      def <<(node)
+        @obj << node
+        self
+      end
+    end
+
+    class ObjectNode
+      attr_reader :obj
+
+      def initialize
+        @obj, @key = {}, nil
+      end
+
+      def <<(node)
+        if @key
+          @obj[@key] = node
+          @key = nil
+        else
+          @key = node 
+        end
+        self
+      end
+    end
+
+  end
+end

data/lib/json/stream/parser.rb

@@ -0,0 +1,430 @@
+# encoding: UTF-8
+
+module JSON
+  module Stream
+
+    class ParserError < RuntimeError; end
+
+    # A streaming JSON parser that generates SAX-like events for
+    # state changes. Use the json gem for small documents. Use this
+    # for huge documents that won't fit in memory.
+    class Parser
+      BUF_SIZE      = 512
+      CONTROL       = /[[:cntrl:]]/
+      WS            = /\s/
+      HEX           = /[0-9a-fA-F]/
+      DIGIT         = /[0-9]/
+      DIGIT_1_9     = /[1-9]/
+      DIGIT_END     = /\d$/
+      TRUE_RE       = /[rue]/
+      FALSE_RE      = /[alse]/
+      NULL_RE       = /[ul]/
+      TRUE_KEYWORD  = 'true'
+      FALSE_KEYWORD = 'false'
+      NULL_KEYWORD  = 'null'
+      LEFT_BRACE    = '{'
+      RIGHT_BRACE   = '}'
+      LEFT_BRACKET  = '['
+      RIGHT_BRACKET = ']'
+      BACKSLASH     = '\\'
+      SLASH         = '/'
+      QUOTE         = '"'
+      COMMA         = ','
+      COLON         = ':'
+      ZERO          = '0'
+      MINUS         = '-'
+      PLUS          = '+'
+      POINT         = '.'
+      EXPONENT      = /[eE]/
+      B,F,N,R,T,U   = %w[b f n r t u]
+
+      # Parses a full JSON document from a String or an IO stream and returns
+      # the parsed object graph. For parsing small JSON documents with small
+      # memory requirements, use the json gem's faster JSON.parse method instead.
+      def self.parse(json)
+        stream = json.is_a?(String) ? StringIO.new(json) : json
+        parser = Parser.new
+        builder = Builder.new(parser)
+        while (buf = stream.read(BUF_SIZE)) != nil
+          parser << buf
+        end
+        raise ParserError, "unexpected eof" unless builder.result
+        builder.result
+      ensure
+        stream.close
+      end
+
+      %w[start_document end_document start_object end_object
+          start_array end_array key value].each do |name|
+
+        define_method(name) do |&block|
+          @listeners[name] << block
+        end
+
+        define_method("notify_#{name}") do |*args|
+          @listeners[name].each do |block|
+            block.call(*args)
+          end
+        end
+        private "notify_#{name}"
+      end
+
+      # Create a new parser with an optional initialization block where
+      # we can register event callbacks. For example:
+      # parser = JSON::Stream::Parser.new do
+      #   start_document { puts "start document" }
+      #   end_document   { puts "end document" }
+      #   start_object   { puts "start object" }
+      #   end_object     { puts "end object" }
+      #   start_array    { puts "start array" }
+      #   end_array      { puts "end array" }
+      #   key            {|k| puts "key: #{k}" }
+      #   value          {|v| puts "value: #{v}" }
+      # end
+      def initialize(&block)
+        @state = :start_document
+        @utf8 = Buffer.new
+        @listeners = Hash.new {|h, k| h[k] = [] }
+        @stack, @unicode, @buf, @pos = [], "", "", -1
+        instance_eval(&block) if block_given?
+      end
+
+      # Pass data into the parser to advance the state machine and
+      # generate callback events. This is well suited for an EventMachine
+      # receive_data loop.
+      def <<(data)
+        (@utf8 << data).each_char do |ch|
+          @pos += 1
+          case @state
+          when :start_document
+            case ch
+            when LEFT_BRACE
+              @state = :start_object
+              @stack.push(:object)
+              notify_start_document
+              notify_start_object
+            when LEFT_BRACKET
+              @state = :start_array
+              @stack.push(:array)
+              notify_start_document
+              notify_start_array
+            when WS
+              # ignore
+            else
+              error("Expected object or array start")
+            end
+          when :start_object
+            case ch
+            when RIGHT_BRACE
+              end_container(:object)
+            when QUOTE
+              @state = :start_string
+              @stack.push(:key)
+            when WS
+              # ignore
+            else
+              error("Expected object key start")
+            end
+          when :start_string
+            case ch
+            when QUOTE
+              if @stack.pop == :string
+                @state = :end_value
+                notify_value(@buf)
+              else # :key
+                @state = :end_key
+                notify_key(@buf)
+              end
+              @buf = ""
+            when BACKSLASH
+              @state = :start_escape
+            when CONTROL
+              error('Control characters must be escaped')
+            else
+              @buf << ch
+            end
+          when :start_escape
+            case ch
+            when QUOTE, BACKSLASH, SLASH
+              @buf << ch
+              @state = :start_string
+            when B
+              @buf << "\b"
+              @state = :start_string
+            when F
+              @buf << "\f"
+              @state = :start_string
+            when N
+              @buf << "\n"
+              @state = :start_string
+            when R
+              @buf << "\r"
+              @state = :start_string
+            when T
+              @buf << "\t"
+              @state = :start_string
+            when U
+              @state = :unicode_escape
+            else
+              error("Expected escaped character")
+            end
+          when :unicode_escape
+            case ch
+            when HEX
+              @unicode << ch
+              if @unicode.size == 4
+                codepoint = @unicode.slice!(0, 4).hex
+                if codepoint >= 0xD800 && codepoint <= 0xDBFF
+                  error('Expected low surrogate pair half') if @stack[-1].is_a?(Fixnum)
+                  @state = :start_surrogate_pair
+                  @stack.push(codepoint)
+                elsif codepoint >= 0xDC00 && codepoint <= 0xDFFF
+                  high = @stack.pop
+                  error('Expected high surrogate pair half') unless high.is_a?(Fixnum)
+                  pair = ((high - 0xD800) * 0x400) + (codepoint - 0xDC00) + 0x10000
+                  @buf << pair
+                  @state = :start_string
+                else
+                  @buf << codepoint
+                  @state = :start_string
+                end
+              end
+            else
+              error('Expected unicode escape hex digit')
+            end
+          when :start_surrogate_pair
+            case ch
+            when BACKSLASH 
+              @state = :start_surrogate_pair_u
+            else
+              error('Expected low surrogate pair half')
+            end
+          when :start_surrogate_pair_u
+            case ch
+            when U
+              @state = :unicode_escape
+            else
+              error('Expected low surrogate pair half')
+            end
+          when :start_negative_number
+            case ch
+            when ZERO
+              @state = :start_zero
+              @buf << ch
+            when DIGIT_1_9
+              @state = :start_int
+              @buf << ch
+            else
+              error('Expected 0-9 digit')
+            end
+          when :start_zero
+            case ch
+            when POINT
+              @state = :start_float
+              @buf << ch
+            when EXPONENT
+              @state = :start_exponent
+              @buf << ch
+            else
+              @state = :end_value
+              notify_value(@buf.to_i)
+              @buf = ""
+              @pos -= 1
+              redo
+            end
+          when :start_float
+            case ch
+            when DIGIT
+              @state = :in_float
+              @buf << ch
+            else
+              error('Expected 0-9 digit')
+            end
+          when :in_float
+            case ch
+            when DIGIT
+              @buf << ch
+            when EXPONENT
+              @state = :start_exponent
+              @buf << ch
+            else
+              @state = :end_value
+              notify_value(@buf.to_f)
+              @buf = ""
+              @pos -= 1
+              redo
+            end
+          when :start_exponent
+            case ch
+            when MINUS, PLUS, DIGIT
+              @state = :in_exponent
+              @buf << ch
+            else
+              error('Expected +, -, or 0-9 digit')
+            end
+          when :in_exponent
+            case ch
+            when DIGIT
+              @buf << ch
+            else
+              error('Expected 0-9 digit') unless @buf =~ DIGIT_END
+              @state = :end_value
+              num = @buf.include?('.') ? @buf.to_f : @buf.to_i
+              notify_value(num)
+              @buf = ""
+              @pos -= 1
+              redo
+            end
+          when :start_int
+            case ch
+            when DIGIT
+              @buf << ch
+            when POINT
+              @state = :start_float
+              @buf << ch
+            when EXPONENT
+              @state = :start_exponent
+              @buf << ch
+            else
+              @state = :end_value
+              notify_value(@buf.to_i)
+              @buf = ""
+              @pos -= 1
+              redo
+            end
+          when :start_true
+            keyword(TRUE_KEYWORD, true, TRUE_RE, ch)
+          when :start_false
+            keyword(FALSE_KEYWORD, false, FALSE_RE, ch)
+          when :start_null
+            keyword(NULL_KEYWORD, nil, NULL_RE, ch)
+          when :end_key
+            case ch
+            when COLON
+              @state = :key_sep
+            when WS
+              # ignore
+            else
+              error("Expected colon key separator")
+            end
+          when :key_sep
+            start_value(ch)
+          when :start_array
+            case ch
+            when RIGHT_BRACKET
+              end_container(:array)
+            when WS
+              # ignore
+            else
+              start_value(ch)
+            end
+          when :end_value
+            case ch
+            when COMMA
+              @state = :value_sep
+            when RIGHT_BRACKET
+              end_container(:array)
+            when RIGHT_BRACE
+              end_container(:object)
+            when WS
+              # ignore
+            else
+              error("Expected comma or object or array close")
+            end
+          when :value_sep
+            if @stack[-1] == :object
+              case ch
+              when QUOTE
+                @state = :start_string
+                @stack.push(:key)
+              when WS
+                # ignore
+              else
+                error("Expected object key start")
+              end
+            else
+              start_value(ch)
+            end
+          when :end_document
+            error("Unexpected data") unless ch =~ WS
+          end
+        end
+      end
+
+      private
+
+      def end_container(type)
+        @state = :end_value
+        if @stack.pop == type
+          send("notify_end_#{type}")
+        else
+          error("Expected end of #{type}")
+        end
+        if @stack.empty?
+          @state = :end_document
+          notify_end_document
+        end
+      end
+
+      def keyword(word, value, re, ch)
+        if ch =~ re
+          @buf << ch
+        else
+          error("Expected #{word} keyword")
+        end
+        if @buf.size == word.size
+          if @buf == word
+            @state = :end_value
+            @buf = ""
+            notify_value(value)
+          else
+            error("Expected #{word} keyword")
+          end
+        end
+      end
+
+      def start_value(ch)
+        case ch
+        when LEFT_BRACE
+          @state = :start_object
+          @stack.push(:object)
+          notify_start_object
+        when LEFT_BRACKET
+          @state = :start_array
+          @stack.push(:array)
+          notify_start_array
+        when QUOTE
+          @state = :start_string
+          @stack.push(:string)
+        when T
+          @state = :start_true
+          @buf << ch
+        when F
+          @state = :start_false
+          @buf << ch
+        when N
+          @state = :start_null
+          @buf << ch
+        when MINUS
+          @state = :start_negative_number
+          @buf << ch
+        when ZERO
+          @state = :start_zero
+          @buf << ch
+        when DIGIT_1_9
+          @state = :start_int
+          @buf << ch
+        when WS
+          # ignore
+        else
+          error("Expected value")
+        end
+      end
+
+      def error(message)
+        raise ParserError, "#{message}: char #{@pos}"
+      end
+    end
+
+  end
+end