oortle-yajl-ruby 0.5.8
Sign up to get free protection for your applications and to get access to all the features.
- data/.gitignore +6 -0
- data/CHANGELOG.md +177 -0
- data/MIT-LICENSE +20 -0
- data/README.rdoc +270 -0
- data/Rakefile +35 -0
- data/VERSION.yml +4 -0
- data/benchmark/encode.rb +46 -0
- data/benchmark/encode_json_and_marshal.rb +35 -0
- data/benchmark/encode_json_and_yaml.rb +47 -0
- data/benchmark/http.rb +30 -0
- data/benchmark/parse.rb +49 -0
- data/benchmark/parse_json_and_marshal.rb +47 -0
- data/benchmark/parse_json_and_yaml.rb +56 -0
- data/benchmark/parse_stream.rb +48 -0
- data/benchmark/subjects/item.json +1 -0
- data/benchmark/subjects/ohai.json +1216 -0
- data/benchmark/subjects/ohai.marshal_dump +0 -0
- data/benchmark/subjects/ohai.yml +975 -0
- data/benchmark/subjects/twitter_search.json +1 -0
- data/benchmark/subjects/twitter_stream.json +430 -0
- data/benchmark/subjects/unicode.json +1 -0
- data/examples/http/twitter_search_api.rb +15 -0
- data/examples/http/twitter_stream_api.rb +27 -0
- data/examples/parsing/from_file.rb +14 -0
- data/examples/parsing/from_stdin.rb +9 -0
- data/examples/parsing/from_string.rb +15 -0
- data/ext/api/yajl_common.h +85 -0
- data/ext/api/yajl_gen.h +123 -0
- data/ext/api/yajl_parse.h +182 -0
- data/ext/extconf.rb +8 -0
- data/ext/yajl.c +157 -0
- data/ext/yajl_alloc.c +65 -0
- data/ext/yajl_alloc.h +50 -0
- data/ext/yajl_buf.c +119 -0
- data/ext/yajl_buf.h +73 -0
- data/ext/yajl_bytestack.h +85 -0
- data/ext/yajl_encode.c +179 -0
- data/ext/yajl_encode.h +44 -0
- data/ext/yajl_ext.c +774 -0
- data/ext/yajl_ext.h +74 -0
- data/ext/yajl_gen.c +290 -0
- data/ext/yajl_lex.c +744 -0
- data/ext/yajl_lex.h +135 -0
- data/ext/yajl_parser.c +447 -0
- data/ext/yajl_parser.h +79 -0
- data/lib/yajl.rb +80 -0
- data/lib/yajl/bzip2.rb +11 -0
- data/lib/yajl/bzip2/stream_reader.rb +29 -0
- data/lib/yajl/bzip2/stream_writer.rb +15 -0
- data/lib/yajl/deflate.rb +6 -0
- data/lib/yajl/deflate/stream_reader.rb +38 -0
- data/lib/yajl/deflate/stream_writer.rb +21 -0
- data/lib/yajl/gzip.rb +6 -0
- data/lib/yajl/gzip/stream_reader.rb +28 -0
- data/lib/yajl/gzip/stream_writer.rb +14 -0
- data/lib/yajl/http_stream.rb +150 -0
- data/lib/yajl/json_gem.rb +14 -0
- data/lib/yajl/json_gem/encoding.rb +40 -0
- data/lib/yajl/json_gem/parsing.rb +26 -0
- data/spec/encoding/encoding_spec.rb +186 -0
- data/spec/http/fixtures/http.bzip2.dump +0 -0
- data/spec/http/fixtures/http.deflate.dump +0 -0
- data/spec/http/fixtures/http.gzip.dump +0 -0
- data/spec/http/fixtures/http.raw.dump +1226 -0
- data/spec/http/http_spec.rb +158 -0
- data/spec/json_gem_compatibility/compatibility_spec.rb +178 -0
- data/spec/parsing/active_support_spec.rb +64 -0
- data/spec/parsing/chunked_spec.rb +98 -0
- data/spec/parsing/fixtures/fail.15.json +1 -0
- data/spec/parsing/fixtures/fail.16.json +1 -0
- data/spec/parsing/fixtures/fail.17.json +1 -0
- data/spec/parsing/fixtures/fail.26.json +1 -0
- data/spec/parsing/fixtures/fail11.json +1 -0
- data/spec/parsing/fixtures/fail12.json +1 -0
- data/spec/parsing/fixtures/fail13.json +1 -0
- data/spec/parsing/fixtures/fail14.json +1 -0
- data/spec/parsing/fixtures/fail19.json +1 -0
- data/spec/parsing/fixtures/fail20.json +1 -0
- data/spec/parsing/fixtures/fail21.json +1 -0
- data/spec/parsing/fixtures/fail22.json +1 -0
- data/spec/parsing/fixtures/fail23.json +1 -0
- data/spec/parsing/fixtures/fail24.json +1 -0
- data/spec/parsing/fixtures/fail25.json +1 -0
- data/spec/parsing/fixtures/fail27.json +2 -0
- data/spec/parsing/fixtures/fail28.json +2 -0
- data/spec/parsing/fixtures/fail3.json +1 -0
- data/spec/parsing/fixtures/fail4.json +1 -0
- data/spec/parsing/fixtures/fail5.json +1 -0
- data/spec/parsing/fixtures/fail6.json +1 -0
- data/spec/parsing/fixtures/fail9.json +1 -0
- data/spec/parsing/fixtures/pass.array.json +6 -0
- data/spec/parsing/fixtures/pass.codepoints_from_unicode_org.json +1 -0
- data/spec/parsing/fixtures/pass.contacts.json +1 -0
- data/spec/parsing/fixtures/pass.db100.xml.json +1 -0
- data/spec/parsing/fixtures/pass.db1000.xml.json +1 -0
- data/spec/parsing/fixtures/pass.dc_simple_with_comments.json +11 -0
- data/spec/parsing/fixtures/pass.deep_arrays.json +1 -0
- data/spec/parsing/fixtures/pass.difficult_json_c_test_case.json +1 -0
- data/spec/parsing/fixtures/pass.difficult_json_c_test_case_with_comments.json +1 -0
- data/spec/parsing/fixtures/pass.doubles.json +1 -0
- data/spec/parsing/fixtures/pass.empty_array.json +1 -0
- data/spec/parsing/fixtures/pass.empty_string.json +1 -0
- data/spec/parsing/fixtures/pass.escaped_bulgarian.json +4 -0
- data/spec/parsing/fixtures/pass.escaped_foobar.json +1 -0
- data/spec/parsing/fixtures/pass.item.json +1 -0
- data/spec/parsing/fixtures/pass.json-org-sample1.json +23 -0
- data/spec/parsing/fixtures/pass.json-org-sample2.json +11 -0
- data/spec/parsing/fixtures/pass.json-org-sample3.json +26 -0
- data/spec/parsing/fixtures/pass.json-org-sample4-nows.json +88 -0
- data/spec/parsing/fixtures/pass.json-org-sample4.json +89 -0
- data/spec/parsing/fixtures/pass.json-org-sample5.json +27 -0
- data/spec/parsing/fixtures/pass.map-spain.xml.json +1 -0
- data/spec/parsing/fixtures/pass.ns-invoice100.xml.json +1 -0
- data/spec/parsing/fixtures/pass.ns-soap.xml.json +1 -0
- data/spec/parsing/fixtures/pass.numbers-fp-4k.json +6 -0
- data/spec/parsing/fixtures/pass.numbers-fp-64k.json +61 -0
- data/spec/parsing/fixtures/pass.numbers-int-4k.json +11 -0
- data/spec/parsing/fixtures/pass.numbers-int-64k.json +154 -0
- data/spec/parsing/fixtures/pass.twitter-search.json +1 -0
- data/spec/parsing/fixtures/pass.twitter-search2.json +1 -0
- data/spec/parsing/fixtures/pass.unicode.json +3315 -0
- data/spec/parsing/fixtures/pass.yelp.json +1 -0
- data/spec/parsing/fixtures/pass1.json +56 -0
- data/spec/parsing/fixtures/pass2.json +1 -0
- data/spec/parsing/fixtures/pass3.json +6 -0
- data/spec/parsing/fixtures_spec.rb +41 -0
- data/spec/parsing/one_off_spec.rb +54 -0
- data/spec/rcov.opts +4 -0
- data/spec/spec.opts +2 -0
- data/spec/spec_helper.rb +6 -0
- data/yajl-ruby.gemspec +179 -0
- metadata +198 -0
data/.gitignore
ADDED
data/CHANGELOG.md
ADDED
@@ -0,0 +1,177 @@
|
|
1
|
+
# Changelog
|
2
|
+
|
3
|
+
## 0.5.7 (June 23rd, 2009)
|
4
|
+
* You can now pass parser options (like :symbolize_keys for example) to Yajl::HttpStream.get
|
5
|
+
* Refactored spec tests a bit, DRYing up the Yajl::HttpStream specs quite a bit.
|
6
|
+
* Added a spec rake task, and spec.opts file
|
7
|
+
* Updated and renamed rcov rake task, and added rcov.opts file
|
8
|
+
|
9
|
+
## 0.5.6 (June 19th, 2009)
|
10
|
+
* Added JSON.default_options hash to the JSON gem compatibility API
|
11
|
+
* Split out the JSON gem compatibility API's parsing and encoding methods into individually includable files
|
12
|
+
** the use case here is if you *only* want parsing, or *only* want encoding
|
13
|
+
** also, if you don't include encoding it won't include the #to_json overrides which tend to cause problems in some environments.
|
14
|
+
* Removed some large benchmark test files to reduce the size of the packaged gem by 1.5MB!
|
15
|
+
|
16
|
+
## 0.5.5 (June 17th, 2009)
|
17
|
+
* Introduction of the JSON gem compatibility API
|
18
|
+
** NOTE: this isn't a 1:1 compatibility API, the goal was to be compatible with as many of the projects using the JSON gem as possible - not the JSON gem API itself
|
19
|
+
** the compatibility API must be explicitly enabled by requiring 'yajl/json_gem' in your project
|
20
|
+
** JSON.parse, JSON.generate, and the #to_json instance method extension to ruby's primitive classes are all included
|
21
|
+
* Fix Yajl::Encoder to ensure map keys are strings
|
22
|
+
* Encoding multiple JSON objects to a single stream doesn't separate by a newline character anymore
|
23
|
+
* Yajl::Encoder now checks for the existence of, and will call #to_json on any non-primitive object
|
24
|
+
|
25
|
+
## 0.5.4 (June 16th, 2009)
|
26
|
+
* Yajl::Parser's :symbolize_keys option now defaults to false
|
27
|
+
* remove use of sprintf for a little speed improvement while parsing
|
28
|
+
|
29
|
+
## 0.5.3 (June 7th, 2009)
|
30
|
+
* The IO parameter for Yajl::Encode#encode is now optional, and accepts a block
|
31
|
+
** it will return the resulting JSON string if no IO is passed to stream to
|
32
|
+
** if a block is passed, it will call and pass it the resulting JSON string
|
33
|
+
* Yajl::Parser#parse can now parse from a String as well as an IO
|
34
|
+
* Added and updated lot of in-code documentation.
|
35
|
+
** all the C code exposed to Ruby should now have comments
|
36
|
+
* Added :symbolize_keys option to the Yajl::Parser class, which defaults to true.
|
37
|
+
** Having this option enabled has shown around an 18% speedup in parsing time according to my benchmarks
|
38
|
+
|
39
|
+
## 0.5.2 (May 30th, 2009)
|
40
|
+
* Added class helper methods Yajl::Encoder.encode(obj, io) and Yajl::Parser.parse(io)
|
41
|
+
* added tests for the above
|
42
|
+
* Updated Twitter streaming example to have a less verbose output
|
43
|
+
* Patch Yajl so encoding can continue as a stream
|
44
|
+
* IE: multiple objects encoded onto the same IO
|
45
|
+
* added a test for the above
|
46
|
+
* Set the internal read buffer size back down to 8kb by default
|
47
|
+
* Added an internal write buffer size (set to 8kb by default) which is used to throttle writes to the output stream
|
48
|
+
* This is to fix a major performance bug/issue with the IO#write C method in ruby 1.9.x (I've opened a bug with them about it)
|
49
|
+
* Fixed a typo in a one-off parsing spec test
|
50
|
+
* Updated benchmarks to work properly in 1.9 (required removal ActiveSupport benchmarking for now)
|
51
|
+
* Updated spec tests to respect ridiculous differences in hash key ordering between 1.8 and 1.9
|
52
|
+
|
53
|
+
|
54
|
+
## 0.5.1 (May 25th, 2009)
|
55
|
+
* added some more tests for the new API
|
56
|
+
* inlined a couple of hot functions used in parsing for a little speedup
|
57
|
+
* updates to readme, reflecting changes in API
|
58
|
+
* version bump to push another gem build
|
59
|
+
|
60
|
+
## 0.5.0 (May 25th, 2009)
|
61
|
+
* Refactored internal API so the caller can specify initialization options for the Parser and Encoder respectively. Two new classes were introduced as a result - Yajl::Parser and Yajl::Encoder. The newly refactored codebase is cleaner, thread-safe and removed all of the hack-code that was trickled around to make things work in the previous implementation. She's much more seaworthy now cap'n!
|
62
|
+
* Yajl::Parser.new accepts two options, :allow_comments and :check_utf8 which both default to true
|
63
|
+
* Yajl::Encoder.new accepts two options, :pretty and :indent which default to false and " " respectively
|
64
|
+
* cleaned up a lot of state code, that to my knowledge prevented yajl-ruby from being used in a thread-safe environment.
|
65
|
+
* added deprecated messaging to Yajl::Stream.parse and Yajl::Stream.encode - these will likely go away before 0.6.0
|
66
|
+
* fixed a bug in the chunked http response parser regarding partially received chunks
|
67
|
+
* added a Twitter Search API example showing off the HttpStream API
|
68
|
+
|
69
|
+
## 0.4.9 (May 20th, 2009)
|
70
|
+
* fixed some parser state bugs surfaced by edge cases
|
71
|
+
* added support for Chunked HTTP response bodies in Yajl::HttpStream
|
72
|
+
* added support for passing a block to Yajl::HttpStream.get that will be used as a callback whenever a JSON object is parsed off the stream (even if there is more than one!)
|
73
|
+
* added an examples folder, and put an example using the Twitter Streaming API in there to start
|
74
|
+
* added some more spec tests, this time around Chunked parsing and continuously parsing multiple JSON strings
|
75
|
+
|
76
|
+
## 0.4.8 (May 18th, 2009)
|
77
|
+
* fixed a totally bone-head compilation problem, I created for myself ;)
|
78
|
+
|
79
|
+
## 0.4.7 (May 18th, 2009)
|
80
|
+
* Bundling Yajl sources to remove the need to install them (and CMake) separately (Thank you Lloyd!!!) This means you can now simply install the gem and be off and running
|
81
|
+
* Added some spec tests for Yajl::HttpStream
|
82
|
+
* Added some spec tests for Yajl::Stream.encode
|
83
|
+
* added some more thank you's, where credit's due - in the readme
|
84
|
+
* updated the unicode.json file to reflect a "real-life" JSON response
|
85
|
+
* reorganized spec tests into their functional areas
|
86
|
+
* added an rcov rake task to generate code coverage output
|
87
|
+
|
88
|
+
## 0.4.6 (May 17th, 2009)
|
89
|
+
* Applied a patch from benburkert (http://github.com/benburkert) to fix HTTP Basic Auth in Yajl::HttpStream.get
|
90
|
+
|
91
|
+
## 0.4.5 (May 17th, 2009)
|
92
|
+
* added Yajl::Stream.encode(hash, io)
|
93
|
+
* generates a JSON string stream, and writes to IO
|
94
|
+
* compressed StreamWriter helpers added as well
|
95
|
+
* fixed a pretty lame segfault in (x86_64 only?) ubuntu/linux
|
96
|
+
* changed the compiled extension to have a more specific name (yajl_ext) for easier loading
|
97
|
+
* removed forced-load of .bundle file, for the rest of the planet aside from OSX users
|
98
|
+
* added some more benchmarks to compare to other forms of serialization in Ruby
|
99
|
+
* various readme updates
|
100
|
+
|
101
|
+
## 0.4.4 (May 12th, 2009)
|
102
|
+
* NOTE: Breaking API change:
|
103
|
+
* renamed Yajl::GzipStreamReader to Yajl::Gzip::StreamReader
|
104
|
+
* added Yajl::Bzip2::StreamReader
|
105
|
+
* depends on the bzip2-ruby gem if you want to use it, if not Yajl::Bzip2 won't be loaded
|
106
|
+
* added Yajl::Deflate::StreamReader
|
107
|
+
* actually uses Zlib::Inflate for stream decompression
|
108
|
+
* added parse(io) class methods to Yajl::Gzip::StreamReader and Yajl::Bzip2::StreamReader as a helper for parsing compressed streams.
|
109
|
+
* updated Yajl::HttpStream to request responses compressed as deflate and bzip2 in addition to gzip
|
110
|
+
* fixed a bug regarding parsing Integers as Floats (so 123456 would have be parsed and returned as 123456.0)
|
111
|
+
* fixed a bug which caused a segfault in ruby's GC during string replacement in Yajl::Gzip and Yajl::Bzip2's StreamReader#read methods
|
112
|
+
* added support for user-specified User-Agent strings in Yajl::HttpStream
|
113
|
+
|
114
|
+
## 0.4.3 (May 2nd, 2009)
|
115
|
+
* adding text/plain as an allowed mime-type for Yajl::HttpStream for webservers that respond with it instead of application/json (ahem...Yelp...)
|
116
|
+
* renamed specs folder to spec for no reason at all
|
117
|
+
|
118
|
+
## 0.4.2 (April 30th, 2009)
|
119
|
+
* Yajl::HttpStream is now sending "proper" http request headers
|
120
|
+
* Yajl::HttpStream will request HTTP-Basic auth if credentials are provided in the passed URI
|
121
|
+
* cleanup requires
|
122
|
+
|
123
|
+
## 0.4.1 (April 30th, 2009)
|
124
|
+
* fixed a typo in the stream.rb benchmark file
|
125
|
+
* fixed a bug in Yajl::Stream.parse that was causing "strange" Ruby malloc errors on large files, with large strings
|
126
|
+
* added Yajl::GzipStreamReader as a wrapper around Zlib::GzipReader to allow for standard IO#read behavior
|
127
|
+
* this allows Yajl::Stream to read off of a Gzip stream directly
|
128
|
+
|
129
|
+
## 0.4.0 (April 29th, 2009)
|
130
|
+
* NOTE: Breaking API change:
|
131
|
+
* refactored Stream parsing methods out of Yajl::Native into Yajl::Stream
|
132
|
+
* removed Yajl::Native namespace/module
|
133
|
+
* Addition of Yajl::HttpStream module
|
134
|
+
* This module is for streaming JSON HTTP responses directly into Yajl (as they're being received) for increased awesomeness
|
135
|
+
* it currently supports basic get requests with Yajl::HttpStream.get(uri)
|
136
|
+
* it also supports (and prefers) output compressed (gzip) responses
|
137
|
+
* Addition Yajl::Chunked module
|
138
|
+
* This module is for feeding Yajl JSON pieces at a time, instead of an entire IO object
|
139
|
+
* This works very well in environments like an EventMachine app where data is received in chunks by design
|
140
|
+
* decreased read buffer for Yajl::Stream from 8kb to 4kb
|
141
|
+
|
142
|
+
## 0.3.4 (April 24th, 2009)
|
143
|
+
* turned Unicode checks back on in the Yajl parser now that it's fixed (thanks Lloyd!)
|
144
|
+
* this also bumps the yajl version dependency requirement to 1.0.4
|
145
|
+
* better guessing of Integer/Float from number found instead of just trying to create a BigNum no matter what
|
146
|
+
* changed extconf.rb to fail Makefile creation if yajl isn't found
|
147
|
+
* added a test to check for parsing Infinity due to a Float overflow
|
148
|
+
|
149
|
+
## 0.3.3 (April 24th, 2009)
|
150
|
+
* 1.9 compatibility
|
151
|
+
|
152
|
+
## 0.3.2 (April 24th, 2009)
|
153
|
+
* version bump: forgot to include yajl.c in the gem
|
154
|
+
|
155
|
+
## 0.3.1 (April 23rd, 2009)
|
156
|
+
* fixed borked gemspec
|
157
|
+
|
158
|
+
## 0.3.0 (April 23rd, 2009)
|
159
|
+
* slight refactor of ActiveSupport tests to better reflect how they actually exist in ActiveSupport
|
160
|
+
* typo correction in the changelog which had the years in 2008
|
161
|
+
* added some initial spec tests
|
162
|
+
* ported some from ActiveSupport to ensure proper compatibility
|
163
|
+
* included 57 JSON fixtures to test against, all of which pass
|
164
|
+
* changed parser config to not check for invalid unicode characters as Ruby is going to do this anyway (?). This resolves the remaining test failures around unicode.
|
165
|
+
* changed how the parser was dealing with numbers to prevent overflows
|
166
|
+
* added an exception class Yajl::ParseError which is now used in place of simply printing to STDERR upon a parsing error
|
167
|
+
* renamed a couple of JSON test files in the benchmark folder to better represent their contents
|
168
|
+
* misc README updates
|
169
|
+
|
170
|
+
## 0.2.1 (April 23rd, 2009)
|
171
|
+
* fixed parsing bug - also fixed failing ActiveSupport test failures (except for the unicode one, which is an issue in Yajl itself)
|
172
|
+
|
173
|
+
## 0.2.0 (April 22nd, 2009)
|
174
|
+
* updated gemspec and README
|
175
|
+
|
176
|
+
## 0.1.0 (April 21st, 2009)
|
177
|
+
* initial release - gemified
|
data/MIT-LICENSE
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright (c) 2008-2009 Brian Lopez - http://github.com/brianmario
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.rdoc
ADDED
@@ -0,0 +1,270 @@
|
|
1
|
+
= YAJL C Bindings for Ruby
|
2
|
+
|
3
|
+
This gem (although not in gem form just yet) is a C binding to the excellent YAJL JSON parsing and generation library.
|
4
|
+
|
5
|
+
You can read more info at the projects website http://lloydforge.org/projects/yajl or check out its codes at http://github.com/lloyd/yajl.
|
6
|
+
|
7
|
+
== Features
|
8
|
+
|
9
|
+
* JSON parsing and encoding directly to and from an IO stream (file, socket, etc) or String. Compressed stream parsing and encoding supported for Bzip2, Gzip and Deflate.
|
10
|
+
* Parse and encode *multiple* JSON objects to and from streams or strings continuously.
|
11
|
+
* JSON gem compatibility API - allows yajl-ruby to be used as a drop-in replacement for the JSON gem
|
12
|
+
* Basic HTTP client (GET and POST) which parses JSON directly off the response body *as it's being received*
|
13
|
+
* ~3.5x faster than JSON.generate
|
14
|
+
* ~1.9x faster than JSON.parse
|
15
|
+
* ~4.5x faster than YAML.load
|
16
|
+
* ~377.5x faster than YAML.dump
|
17
|
+
* ~1.5x faster than Marshal.load
|
18
|
+
* ~2x faster than Marshal.dump
|
19
|
+
|
20
|
+
== How to install
|
21
|
+
|
22
|
+
Install it like any other gem hosted at the Githubs like so:
|
23
|
+
|
24
|
+
(more instructions here: http://gems.github.com)
|
25
|
+
|
26
|
+
sudo gem install brianmario-yajl-ruby
|
27
|
+
|
28
|
+
== Example of use
|
29
|
+
|
30
|
+
First, you're probably gonna want to require it:
|
31
|
+
|
32
|
+
require 'yajl'
|
33
|
+
|
34
|
+
=== Parsing
|
35
|
+
|
36
|
+
Then maybe parse some JSON from:
|
37
|
+
|
38
|
+
a File IO
|
39
|
+
|
40
|
+
json = File.new('test.json', 'r')
|
41
|
+
parser = Yajl::Parser.new
|
42
|
+
hash = parser.parse(json)
|
43
|
+
|
44
|
+
or maybe a StringIO
|
45
|
+
|
46
|
+
json = StringIO.new("...some JSON...")
|
47
|
+
parser = Yajl::Parser.new
|
48
|
+
hash = parser.parse(json)
|
49
|
+
|
50
|
+
or maybe STDIN
|
51
|
+
|
52
|
+
cat someJsonFile.json | ruby -ryajl -e "puts Yajl::Parser.new.parse(STDIN).inspect"
|
53
|
+
|
54
|
+
|
55
|
+
Or lets say you didn't have access to the IO object that contained JSON data, but instead
|
56
|
+
only had access to chunks of it at a time. No problem!
|
57
|
+
|
58
|
+
(Assume we're in an EventMachine::Connection instance)
|
59
|
+
|
60
|
+
def post_init
|
61
|
+
@parser = Yajl::Parser.new
|
62
|
+
end
|
63
|
+
|
64
|
+
def object_parsed(obj)
|
65
|
+
puts "Sometimes one pays most for the things one gets for nothing. - Albert Einstein"
|
66
|
+
puts obj.inspect
|
67
|
+
end
|
68
|
+
|
69
|
+
def connection_completed
|
70
|
+
# once a full JSON object has been parsed from the stream
|
71
|
+
# object_parsed will be called, and passed the constructed object
|
72
|
+
@parser.on_parse_complete = method(:object_parsed)
|
73
|
+
end
|
74
|
+
|
75
|
+
def receive_data(data)
|
76
|
+
# continue passing chunks
|
77
|
+
@parser << data
|
78
|
+
end
|
79
|
+
|
80
|
+
|
81
|
+
Or how about a JSON API HTTP request?
|
82
|
+
This actually makes a request using a raw TCPSocket, then parses the JSON body right off the socket. While it's being received over the wire!
|
83
|
+
|
84
|
+
require 'uri'
|
85
|
+
require 'yajl/http_stream'
|
86
|
+
|
87
|
+
url = URI.parse("http://search.twitter.com/search.json?q=engineyard")
|
88
|
+
results = Yajl::HttpStream.get(url)
|
89
|
+
|
90
|
+
Or do the same request, with Gzip and Deflate output compression support (also supports Bzip2, if loaded):
|
91
|
+
(this does the same raw socket Request, but transparently parses the compressed response body)
|
92
|
+
|
93
|
+
require 'uri'
|
94
|
+
require 'yajl/gzip'
|
95
|
+
require 'yajl/deflate'
|
96
|
+
require 'yajl/http_stream'
|
97
|
+
|
98
|
+
url = URI.parse("http://search.twitter.com/search.json?q=engineyard")
|
99
|
+
results = Yajl::HttpStream.get(url)
|
100
|
+
|
101
|
+
Since yajl-ruby parses JSON as a stream, supporting API's like Twitter's Streaming API are a piece-of-cake.
|
102
|
+
You can simply supply a block to Yajl::HttpStream.get, which is used as the callback for when a JSON object has been
|
103
|
+
unserialized off the stream. For the case of this Twitter Streaming API call, the callback gets fired a few times a second (depending on your connection speed).
|
104
|
+
The code below is all that's needed to make the request and stream unserialized Ruby hashes off the response, continuously.
|
105
|
+
You'll note that I've enabled the :symbolize_keys parser option as well. Doing so is much more efficient for parsing JSON streams with
|
106
|
+
lots of repetitive keys - for things like result sets or multiple API requests - than the same parse with string keys.
|
107
|
+
This is because Ruby will reuse (and never GC) it's symbol table. Be that as it may, if you want to parse JSON strings with random key names
|
108
|
+
it's much better to leave string keys enabled (the default), so they can get GC'd later.
|
109
|
+
|
110
|
+
require 'uri'
|
111
|
+
require 'yajl/http_stream'
|
112
|
+
|
113
|
+
uri = URI.parse("http://#{username}:#{password}@stream.twitter.com/spritzer.json")
|
114
|
+
Yajl::HttpStream.get(uri, :symbolize_keys => true) do |hash|
|
115
|
+
puts hash.inspect
|
116
|
+
end
|
117
|
+
|
118
|
+
Or how about parsing directly from a compressed file?
|
119
|
+
|
120
|
+
require 'yajl/bzip2'
|
121
|
+
|
122
|
+
file = File.new('some.json.bz2', 'r')
|
123
|
+
result = Yajl::Bzip2::StreamReader.parse(file)
|
124
|
+
|
125
|
+
=== Encoding
|
126
|
+
|
127
|
+
Since yajl-ruby does everything using streams, you simply need to pass the object to encode, and the IO to write the stream to (this happens in chunks).
|
128
|
+
|
129
|
+
This allows you to encode JSON as a stream, writing directly to a socket
|
130
|
+
|
131
|
+
socket = TCPSocket.new(192.168.1.101, 9000)
|
132
|
+
hash = {:foo => 12425125, :bar => "some string", ... }
|
133
|
+
encoder = Yajl::Encoder.new
|
134
|
+
Yajl::Encoder.encode(hash, socket)
|
135
|
+
|
136
|
+
Or what if you wanted to compress the stream over the wire?
|
137
|
+
|
138
|
+
require 'yajl/gzip'
|
139
|
+
socket = TCPSocket.new(192.168.1.101, 9000)
|
140
|
+
hash = {:foo => 12425125, :bar => "some string", ... }
|
141
|
+
Yajl::Gzip::StreamWriter.encode(hash, socket)
|
142
|
+
|
143
|
+
Or what about encoding multiple objects to JSON over the same stream?
|
144
|
+
This example will encode and send 50 JSON objects over the same stream, continuously.
|
145
|
+
|
146
|
+
socket = TCPSocket.new(192.168.1.101, 9000)
|
147
|
+
encoder = Yajl::Encoder.new
|
148
|
+
50.times do
|
149
|
+
hash = {:current_time => Time.now.to_f, :foo => 12425125}
|
150
|
+
encoder.encode(hash, socket)
|
151
|
+
end
|
152
|
+
|
153
|
+
You can also use Yajl::Bzip2::StreamWriter and Yajl::Deflate::StreamWriter. So you can pick whichever fits your CPU/bandwidth sweet-spot.
|
154
|
+
|
155
|
+
== JSON gem Compatibility API
|
156
|
+
|
157
|
+
The JSON gem compatibility API isn't enabled by default. You have to explicitly require it like so:
|
158
|
+
|
159
|
+
require 'yajl/json_gem'
|
160
|
+
|
161
|
+
That's right, you can just replace "require 'json'" with the line above and you're done!
|
162
|
+
|
163
|
+
This will require yajl-ruby itself, as well as enable it's JSON gem compatibility API.
|
164
|
+
|
165
|
+
This includes the following API:
|
166
|
+
|
167
|
+
JSON.parse, JSON.generate, JSON.pretty_generate, JSON.load, JSON.dump
|
168
|
+
and all of the #to_json instance method overrides for Ruby's primitive objects
|
169
|
+
|
170
|
+
Once the compatibility API is enabled, your existing or new project should work as if the JSON gem itself were being used. Only you'll be using Yajl ;)
|
171
|
+
|
172
|
+
There are a lot more possibilities that I'd love to see other gems/plugins for someday.
|
173
|
+
|
174
|
+
Some ideas are:
|
175
|
+
* parsing logs in JSON format
|
176
|
+
* a Rails plugin (http://github.com/technoweenie/yajl-rails)
|
177
|
+
* builtin Rails 3 support?
|
178
|
+
* Rack middleware (ideally the JSON body could be handed to the parser while it's still being received)
|
179
|
+
* use with ohai (http://github.com/brianmario/ohai)
|
180
|
+
* JSON API clients (http://github.com/brianmario/crack, http://github.com/brianmario/freckle-api)
|
181
|
+
* Patch Marshal#load and Marshal#dump to use JSON? ;)
|
182
|
+
* etc...
|
183
|
+
|
184
|
+
== Benchmarks
|
185
|
+
|
186
|
+
After I finished implementation - this library performs close to the same as the current JSON.parse (C gem) does on small/medium files.
|
187
|
+
|
188
|
+
But on larger files, and higher amounts of iteration, this library was around 2x faster than JSON.parse.
|
189
|
+
|
190
|
+
The main benefit of this library is in it's memory usage.
|
191
|
+
Since it's able to parse the stream in chunks, it's memory requirements are very, very low.
|
192
|
+
|
193
|
+
Here's what parsing a 2.43MB JSON file off the filesystem 20 times looks like:
|
194
|
+
|
195
|
+
=== Memory Usage
|
196
|
+
|
197
|
+
==== Average
|
198
|
+
|
199
|
+
* Yajl::Parser#parse: 32MB
|
200
|
+
* JSON.parse: 54MB
|
201
|
+
* ActiveSupport::JSON.decode: 63MB
|
202
|
+
|
203
|
+
==== Peak
|
204
|
+
|
205
|
+
* Yajl::Parser#parse: 32MB
|
206
|
+
* JSON.parse: 57MB
|
207
|
+
* ActiveSupport::JSON.decode: 67MB
|
208
|
+
|
209
|
+
=== Parse Time
|
210
|
+
|
211
|
+
* Yajl::Parser#parse: 4.54s
|
212
|
+
* JSON.parse: 5.47s
|
213
|
+
* ActiveSupport::JSON.decode: 64.42s
|
214
|
+
|
215
|
+
=== Encode Time
|
216
|
+
|
217
|
+
* Yajl::Encoder#encode: 3.59s
|
218
|
+
* JSON#to_json: 6.2s
|
219
|
+
* ActiveSupport::JSON.encode: 45.58s
|
220
|
+
|
221
|
+
=== Compared to YAML
|
222
|
+
|
223
|
+
NOTE: I converted the 2.4MB JSON file to YAML for this test.
|
224
|
+
|
225
|
+
==== Parse Time (from their respective formats)
|
226
|
+
|
227
|
+
* Yajl::Parser#parse: 4.33s
|
228
|
+
* JSON.parse: 5.37s
|
229
|
+
* YAML.load: 19.47s
|
230
|
+
|
231
|
+
==== Encode Time (to their respective formats)
|
232
|
+
|
233
|
+
* Yajl::Encoder#encode: 3.47s
|
234
|
+
* JSON#to_json: 6.6s
|
235
|
+
* YAML.dump(obj, io): 1309.93s
|
236
|
+
|
237
|
+
=== Compared to Marshal.load/Marshal.dump
|
238
|
+
|
239
|
+
NOTE: I converted the 2.4MB JSON file to a Hash and a dump file from Marshal.dump for this test.
|
240
|
+
|
241
|
+
==== Parse Time (from their respective formats)
|
242
|
+
|
243
|
+
* Yajl::Parser#parse: 4.54s
|
244
|
+
* JSON.parse: 7.40s
|
245
|
+
* Marshal.load: 7s
|
246
|
+
|
247
|
+
==== Encode Time (to their respective formats)
|
248
|
+
|
249
|
+
* Yajl::Encoder#encode: 2.39s
|
250
|
+
* JSON#to_json: 8.37s
|
251
|
+
* Marshal.dump: 4.66s
|
252
|
+
|
253
|
+
== Third Party Sources Bundled
|
254
|
+
|
255
|
+
This project includes code from the BSD licensed yajl project, copyright 2007-2009 Lloyd Hilaiel
|
256
|
+
|
257
|
+
== Special Thanks
|
258
|
+
|
259
|
+
For those of you using yajl-ruby out in the wild, please hit me up on Twitter (brianmario) or send me a message here on the Githubs describing the site and how you're using it. I'd love to get a list going!
|
260
|
+
|
261
|
+
I've had a lot of inspiration, and a lot of help. Thanks to everyone who's been a part of this and those to come!
|
262
|
+
|
263
|
+
* Lloyd Hilaiel - http://github.com/lloyd - for writing Yajl!!
|
264
|
+
* Josh Ferguson - http://github.com/besquared - for peer-pressuring me into getting back into C; it worked ;) Also tons of support over IM
|
265
|
+
* Jonathan Novak - http://github.com/cypriss - pointer-hacking help
|
266
|
+
* Tom Smith - http://github.com/rtomsmith - pointer-hacking help
|
267
|
+
* Rick http://github.com/technoweenie - for making an ActiveSupport patch with support for this library and teasing me that it might go into Rails 3. You sure lit a fire under my ass and I got a ton of work done because of it! :)
|
268
|
+
* The entire Github Crew - http://github.com/ - my inspiration, time spent writing this, finding Yajl, So many-MANY other things wouldn't have been possible without this awesome service. I owe you guys some whiskey at Kilowatt.
|
269
|
+
* benburkert - http://github.com/benburkert
|
270
|
+
* Aman Gupta - http://github.com/tmm1 - tons of suggestions and inspiration for the most recent features, and hopefully more to come ;)
|