yajl-ruby 0.5.5

data/.gitignore

@@ -0,0 +1,5 @@
+Makefile
+*.o
+*.dylib
+*.bundle
+TODO.txt

data/CHANGELOG.md

@@ -0,0 +1,164 @@
+# Changelog
+
+## 0.5.5 (June 17th, 2009)
+* Introduction of the JSON gem API compatibility layer
+** NOTE: this isn't a 1:1 compatibility API, the goal was to be compatible with as many of the projects using the JSON gem as possible - not the JSON gem API itself
+** the compatibility API must be explicitly enabled by requiring 'yajl/json_gem' in your project
+** JSON.parse, JSON.generate, and the #to_json instance method extension to ruby's primitive classes are all included
+* Fix Yajl::Encoder to ensure map keys are strings
+* Encoding multiple JSON objects to a single stream doesn't separate by a newline character anymore
+* Yajl::Encoder now checks for the existence of, and will call #to_json on any non-primitive object
+
+## 0.5.4 (June 16th, 2009)
+* Yajl::Parser's :symbolize_keys option now defaults to false
+* remove use of sprintf for a little speed improvement while parsing
+
+## 0.5.3 (June 7th, 2009)
+* The IO parameter for Yajl::Encode#encode is now optional, and accepts a block
+** it will return the resulting JSON string if no IO is passed to stream to
+** if a block is passed, it will call and pass it the resulting JSON string
+* Yajl::Parser#parse can now parse from a String as well as an IO
+* Added and updated lot of in-code documentation.
+** all the C code exposed to Ruby should now have comments
+* Added :symbolize_keys option to the Yajl::Parser class, which defaults to true.
+** Having this option enabled has shown around an 18% speedup in parsing time according to my benchmarks
+
+## 0.5.2 (May 30th, 2009)
+* Added class helper methods Yajl::Encoder.encode(obj, io) and Yajl::Parser.parse(io)
+ * added tests for the above
+* Updated Twitter streaming example to have a less verbose output
+* Patch Yajl so encoding can continue as a stream
+ * IE: multiple objects encoded onto the same IO
+ * added a test for the above
+* Set the internal read buffer size back down to 8kb by default
+* Added an internal write buffer size (set to 8kb by default) which is used to throttle writes to the output stream
+ * This is to fix a major performance bug/issue with the IO#write C method in ruby 1.9.x (I've opened a bug with them about it)
+* Fixed a typo in a one-off parsing spec test
+* Updated benchmarks to work properly in 1.9 (required removal ActiveSupport benchmarking for now)
+* Updated spec tests to respect ridiculous differences in hash key ordering between 1.8 and 1.9
+
+
+## 0.5.1 (May 25th, 2009)
+* added some more tests for the new API
+* inlined a couple of hot functions used in parsing for a little speedup
+* updates to readme, reflecting changes in API
+* version bump to push another gem build
+
+## 0.5.0 (May 25th, 2009)
+* Refactored internal API so the caller can specify initialization options for the Parser and Encoder respectively. Two new classes were introduced as a result - Yajl::Parser and Yajl::Encoder. The newly refactored codebase is cleaner, thread-safe and removed all of the hack-code that was trickled around to make things work in the previous implementation. She's much more seaworthy now cap'n!
+ * Yajl::Parser.new accepts two options, :allow_comments and :check_utf8 which both default to true
+ * Yajl::Encoder.new accepts two options, :pretty and :indent which default to false and "  " respectively
+* cleaned up a lot of state code, that to my knowledge prevented yajl-ruby from being used in a thread-safe environment.
+* added deprecated messaging to Yajl::Stream.parse and Yajl::Stream.encode - these will likely go away before 0.6.0
+* fixed a bug in the chunked http response parser regarding partially received chunks
+* added a Twitter Search API example showing off the HttpStream API
+
+## 0.4.9 (May 20th, 2009)
+* fixed some parser state bugs surfaced by edge cases
+* added support for Chunked HTTP response bodies in Yajl::HttpStream
+ * added support for passing a block to Yajl::HttpStream.get that will be used as a callback whenever a JSON object is parsed off the stream (even if there is more than one!)
+* added an examples folder, and put an example using the Twitter Streaming API in there to start
+* added some more spec tests, this time around Chunked parsing and continuously parsing multiple JSON strings
+
+## 0.4.8 (May 18th, 2009)
+* fixed a totally bone-head compilation problem, I created for myself ;)
+
+## 0.4.7 (May 18th, 2009)
+* Bundling Yajl sources to remove the need to install them (and CMake) separately (Thank you Lloyd!!!) This means you can now simply install the gem and be off and running
+* Added some spec tests for Yajl::HttpStream
+* Added some spec tests for Yajl::Stream.encode
+* added some more thank you's, where credit's due - in the readme
+* updated the unicode.json file to reflect a "real-life" JSON response
+* reorganized spec tests into their functional areas
+* added an rcov rake task to generate code coverage output
+
+## 0.4.6 (May 17th, 2009)
+* Applied a patch from benburkert (http://github.com/benburkert) to fix HTTP Basic Auth in Yajl::HttpStream.get
+
+## 0.4.5 (May 17th, 2009)
+* added Yajl::Stream.encode(hash, io)
+ * generates a JSON string stream, and writes to IO
+ * compressed StreamWriter helpers added as well
+* fixed a pretty lame segfault in (x86_64 only?) ubuntu/linux
+* changed the compiled extension to have a more specific name (yajl_ext) for easier loading
+* removed forced-load of .bundle file, for the rest of the planet aside from OSX users
+* added some more benchmarks to compare to other forms of serialization in Ruby
+* various readme updates
+
+## 0.4.4 (May 12th, 2009)
+* NOTE: Breaking API change:
+ * renamed Yajl::GzipStreamReader to Yajl::Gzip::StreamReader
+* added Yajl::Bzip2::StreamReader
+ * depends on the bzip2-ruby gem if you want to use it, if not Yajl::Bzip2 won't be loaded
+* added Yajl::Deflate::StreamReader
+ * actually uses Zlib::Inflate for stream decompression
+* added parse(io) class methods to Yajl::Gzip::StreamReader and Yajl::Bzip2::StreamReader as a helper for parsing compressed streams.
+* updated Yajl::HttpStream to request responses compressed as deflate and bzip2 in addition to gzip
+* fixed a bug regarding parsing Integers as Floats (so 123456 would have be parsed and returned as 123456.0)
+* fixed a bug which caused a segfault in ruby's GC during string replacement in Yajl::Gzip and Yajl::Bzip2's StreamReader#read methods
+* added support for user-specified User-Agent strings in Yajl::HttpStream
+
+## 0.4.3 (May 2nd, 2009)
+* adding text/plain as an allowed mime-type for Yajl::HttpStream for webservers that respond with it instead of application/json (ahem...Yelp...)
+* renamed specs folder to spec for no reason at all
+
+## 0.4.2 (April 30th, 2009)
+* Yajl::HttpStream is now sending "proper" http request headers
+* Yajl::HttpStream will request HTTP-Basic auth if credentials are provided in the passed URI
+* cleanup requires
+
+## 0.4.1 (April 30th, 2009)
+* fixed a typo in the stream.rb benchmark file
+* fixed a bug in Yajl::Stream.parse that was causing "strange" Ruby malloc errors on large files, with large strings
+* added Yajl::GzipStreamReader as a wrapper around Zlib::GzipReader to allow for standard IO#read behavior
+ * this allows Yajl::Stream to read off of a Gzip stream directly
+ 
+## 0.4.0 (April 29th, 2009)
+* NOTE: Breaking API change:
+ * refactored Stream parsing methods out of Yajl::Native into Yajl::Stream
+ * removed Yajl::Native namespace/module
+* Addition of Yajl::HttpStream module
+ * This module is for streaming JSON HTTP responses directly into Yajl (as they're being received) for increased awesomeness
+ * it currently supports basic get requests with Yajl::HttpStream.get(uri)
+ * it also supports (and prefers) output compressed (gzip) responses
+* Addition Yajl::Chunked module
+ * This module is for feeding Yajl JSON pieces at a time, instead of an entire IO object
+ * This works very well in environments like an EventMachine app where data is received in chunks by design
+* decreased read buffer for Yajl::Stream from 8kb to 4kb
+
+## 0.3.4 (April 24th, 2009)
+* turned Unicode checks back on in the Yajl parser now that it's fixed (thanks Lloyd!)
+ * this also bumps the yajl version dependency requirement to 1.0.4
+* better guessing of Integer/Float from number found instead of just trying to create a BigNum no matter what
+* changed extconf.rb to fail Makefile creation if yajl isn't found
+* added a test to check for parsing Infinity due to a Float overflow
+
+## 0.3.3 (April 24th, 2009)
+* 1.9 compatibility
+
+## 0.3.2 (April 24th, 2009)
+* version bump: forgot to include yajl.c in the gem
+
+## 0.3.1 (April 23rd, 2009)
+* fixed borked gemspec
+
+## 0.3.0 (April 23rd, 2009)
+* slight refactor of ActiveSupport tests to better reflect how they actually exist in ActiveSupport
+* typo correction in the changelog which had the years in 2008
+* added some initial spec tests
+ * ported some from ActiveSupport to ensure proper compatibility
+ * included 57 JSON fixtures to test against, all of which pass
+* changed parser config to not check for invalid unicode characters as Ruby is going to do this anyway (?). This resolves the remaining test failures around unicode.
+* changed how the parser was dealing with numbers to prevent overflows
+* added an exception class Yajl::ParseError which is now used in place of simply printing to STDERR upon a parsing error
+* renamed a couple of JSON test files in the benchmark folder to better represent their contents
+* misc README updates
+
+## 0.2.1 (April 23rd, 2009)
+* fixed parsing bug - also fixed failing ActiveSupport test failures (except for the unicode one, which is an issue in Yajl itself)
+
+## 0.2.0 (April 22nd, 2009)
+* updated gemspec and README
+
+## 0.1.0 (April 21st, 2009)
+* initial release - gemified

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2008-2009 Brian Lopez - http://github.com/brianmario
+ 
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+ 
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+ 
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,264 @@
+= YAJL C Bindings for Ruby
+
+This gem (although not in gem form just yet) is a C binding to the excellent YAJL JSON parsing and generation library.
+
+You can read more info at the projects website http://lloydforge.org/projects/yajl or check out it's codes at http://github.com/lloyd/yajl.
+
+== Features
+
+* JSON parsing and encoding directly to and from an IO stream (file, socket, etc) or String. Compressed stream parsing and encoding supported for Bzip2, Gzip and Deflate.
+* Parse and encode *multiple* JSON objects to and from streams or strings continuously.
+* JSON gem compatibility API - allows yajl-ruby to be used as a drop-in replacement for the JSON gem
+* Basic HTTP client (only GET requests supported for now) which parses JSON directly off the response body *as it's being received*
+* ~3.5x faster than JSON.generate
+* ~1.9x faster than JSON.parse
+* ~4.5x faster than YAML.load
+* ~377.5x faster than YAML.dump
+* ~1.5x faster than Marshal.load
+* ~2x faster than Marshal.dump
+
+== How to install
+
+Install it like any other gem hosted at the Githubs like so:
+
+(more instructions here: http://gems.github.com)
+
+ sudo gem install brianmario-yajl-ruby
+
+== Example of use
+
+First, you're probably gonna want to require it:
+
+ require 'yajl'
+
+=== Parsing
+
+Then maybe parse some JSON from:
+
+a File IO
+
+ json = File.new('test.json', 'r')
+ parser = Yajl::Parser.new
+ hash = parser.parse(json)
+
+or maybe a StringIO
+
+ json = StringIO.new("...some JSON...")
+ parser = Yajl::Parser.new
+ hash = parser.parse(json)
+
+or maybe STDIN
+
+ cat someJsonFile.json | ruby -ryajl -e "puts Yajl::Parser.new.parse(STDIN).inspect"
+
+
+Or lets say you didn't have access to the IO object that contained JSON data, but instead
+only had access to chunks of it at a time. No problem!
+
+(Assume we're in an EventMachine::Connection instance)
+
+ def post_init
+    @parser = Yajl::Parser.new
+ end
+ 
+ def object_parsed(obj)
+    puts "Sometimes one pays most for the things one gets for nothing. - Albert Einstein"
+    puts obj.inspect
+  end
+  
+ def connection_completed
+   # once a full JSON object has been parsed from the stream
+   # object_parsed will be called, and passed the constructed object
+   @parser.on_parse_complete = method(:object_parsed)
+ end
+ 
+ def receive_data(data)
+   # continue passing chunks
+   @parser << data
+ end
+
+
+Or how about a JSON API HTTP request?
+This actually makes a request using a raw TCPSocket, then parses the JSON body right off the socket. While it's being received over the wire!
+
+ require 'uri'
+ require 'yajl/http_stream'
+ 
+ url = URI.parse("http://search.twitter.com/search.json?q=engineyard")
+ results = Yajl::HttpStream.get(url)
+
+Or do the same request, with Gzip and Deflate output compression support (also supports Bzip2, if loaded):
+(this does the same raw socket Request, but transparently parses the compressed response body)
+ 
+ require 'uri'
+ require 'yajl/gzip'
+ require 'yajl/deflate'
+ require 'yajl/http_stream'
+
+ url = URI.parse("http://search.twitter.com/search.json?q=engineyard")
+ results = Yajl::HttpStream.get(url)
+
+Since yajl-ruby parses JSON as a stream, supporting API's like Twitter's Streaming API are a piece-of-cake.
+You can simply supply a block to Yajl::HttpStream.get, which is used as the callback for when a JSON object has been
+unserialized off the stream. For the case of this Twitter Streaming API call, the callback gets fired a few times a second (depending on your connection speed).
+The code below is all that's needed to make the request and stream unserialized Ruby hashes off the response, continuously.
+
+ require 'uri'
+ require 'yajl/http_stream'
+ 
+ uri = URI.parse("http://#{username}:#{password}@stream.twitter.com/spritzer.json")
+ Yajl::HttpStream.get(uri) do |hash|
+   puts hash.inspect
+ end
+
+Or how about parsing directly from a compressed file?
+
+ require 'yajl/bzip2'
+ 
+ file = File.new('some.json.bz2', 'r')
+ result = Yajl::Bzip2::StreamReader.parse(file)
+
+=== Encoding
+
+Since yajl-ruby does everything using streams, you simply need to pass the object to encode, and the IO to write the stream to (this happens in chunks).
+
+This allows you to encode JSON as a stream, writing directly to a socket
+
+ socket = TCPSocket.new(192.168.1.101, 9000)
+ hash = {:foo => 12425125, :bar => "some string", ... }
+ encoder = Yajl::Encoder.new
+ Yajl::Encoder.encode(hash, socket)
+
+Or what if you wanted to compress the stream over the wire?
+
+ require 'yajl/gzip'
+ socket = TCPSocket.new(192.168.1.101, 9000)
+ hash = {:foo => 12425125, :bar => "some string", ... }
+ Yajl::Gzip::StreamWriter.encode(hash, socket)
+
+Or what about encoding multiple objects to JSON over the same stream?
+This example will encode and send 50 JSON objects over the same stream, continuously.
+
+ socket = TCPSocket.new(192.168.1.101, 9000)
+ encoder = Yajl::Encoder.new
+ 50.times do
+   hash = {:current_time => Time.now.to_f, :foo => 12425125}
+   encoder.encode(hash, socket)
+ end
+
+You can also use Yajl::Bzip2::StreamWriter and Yajl::Deflate::StreamWriter. So you can pick whichever fits your CPU/bandwidth sweet-spot.
+
+== JSON gem Compatibility API
+
+The JSON gem compatibility API isn't enabled by default. You have to explicitly require it like so:
+
+ require 'yajl/json_gem'
+
+That's right, you can just replace "require 'json'" with the line above and you're done!
+
+This will require yajl-ruby itself, as well as enable it's JSON gem compatibility API.
+
+This includes the following API:
+
+ JSON.parse, JSON.generate, JSON.pretty_generate, JSON.load, JSON.dump
+ and all of the #to_json instance method overrides for Ruby's primitive objects
+
+Once the compatibility API is enabled, you're existing or new project should work as if the JSON gem itself were being used. Only you'll be using Yajl ;)
+
+There are a lot more possibilities that I'd love to see other gems/plugins for someday.
+
+Some ideas are:
+* parsing logs in JSON format
+* a Rails plugin (http://github.com/technoweenie/yajl-rails)
+* builtin Rails 3 support?
+* Rack middleware (ideally the JSON body could be handed to the parser while it's still being received)
+* use with ohai (http://github.com/brianmario/ohai)
+* JSON API clients (http://github.com/brianmario/crack, http://github.com/brianmario/freckle-api)
+* Patch Marshal#load and Marshal#dump to use JSON? ;)
+* etc...
+
+== Benchmarks
+
+After I finished implementation - this library performs close to the same as the current JSON.parse (C gem) does on small/medium files.
+
+But on larger files, and higher amounts of iteration, this library was around 2x faster than JSON.parse.
+
+The main benefit of this library is in it's memory usage.
+Since it's able to parse the stream in chunks, it's memory requirements are very, very low.
+
+Here's what parsing a 2.43MB JSON file off the filesystem 20 times looks like:
+
+=== Memory Usage
+
+==== Average
+
+* Yajl::Parser#parse: 32MB
+* JSON.parse: 54MB
+* ActiveSupport::JSON.decode: 63MB
+
+==== Peak
+
+* Yajl::Parser#parse: 32MB
+* JSON.parse: 57MB
+* ActiveSupport::JSON.decode: 67MB
+
+=== Parse Time
+
+* Yajl::Parser#parse: 4.54s
+* JSON.parse: 5.47s
+* ActiveSupport::JSON.decode: 64.42s
+
+=== Encode Time
+
+* Yajl::Encoder#encode: 3.59s
+* JSON#to_json: 6.2s
+* ActiveSupport::JSON.encode: 45.58s
+
+=== Compared to YAML
+
+NOTE: I converted the 2.4MB JSON file to YAML for this test.
+
+==== Parse Time (from their respective formats)
+
+* Yajl::Parser#parse: 4.33s
+* JSON.parse: 5.37s
+* YAML.load: 19.47s
+
+==== Encode Time (to their respective formats)
+
+* Yajl::Encoder#encode: 3.47s
+* JSON#to_json: 6.6s
+* YAML.dump(obj, io): 1309.93s
+
+=== Compared to Marshal.load/Marshal.dump
+
+NOTE: I converted the 2.4MB JSON file to a Hash and a dump file from Marshal.dump for this test.
+
+==== Parse Time (from their respective formats)
+
+* Yajl::Parser#parse: 4.54s
+* JSON.parse: 7.40s
+* Marshal.load: 7s
+
+==== Encode Time (to their respective formats)
+
+* Yajl::Encoder#encode: 2.39s
+* JSON#to_json: 8.37s
+* Marshal.dump: 4.66s
+
+== Third Party Sources Bundled
+
+This project includes code from the BSD licensed yajl project, copyright 2007-2009 Lloyd Hilaiel
+
+== Special Thanks
+
+I've had a lot of inspiration, and a lot of help. Thanks to everyone who's been a part of this and those to come!
+
+* Lloyd Hilaiel - http://github.com/lloyd - for writing Yajl!!
+* Josh Ferguson - http://github.com/besquared - for peer-pressuring me into getting back into C; it worked ;) Also tons of support over IM
+* Jonathan Novak - http://github.com/cypriss - pointer-hacking help
+* Tom Smith - http://github.com/rtomsmith - pointer-hacking help
+* Rick http://github.com/technoweenie - for making an ActiveSupport patch with support for this library and teasing me that it might go into Rails 3. You sure lit a fire under my ass and I got a ton of work done because of it! :)
+* The entire Github Crew - http://github.com/ - my inspiration, time spent writing this, finding Yajl, So many-MANY other things wouldn't have been possible without this awesome service. I owe you guys some whiskey at Kilowatt.
+* benburkert - http://github.com/benburkert
+* Aman Gupta - http://github.com/tmm1 - tons of suggestions and inspiration for the most recent features, and hopefully more to come ;)