oj 0.5.2 → 0.6.0

data/README.md

@@ -18,13 +18,17 @@ A fast JSON parser and Object marshaller as a Ruby gem.
 
 ## <a name="release">Release Notes</a>
 
-### Release 0.5.2
+### Release 0.6.0
 
-- Release 0.5.2 fixes encoding and float encoding.
+- supports arbitrary Object dumping/serialization
 
-This is the first release sith a version of 0.5 indicating it is only half
-done. Basic load() and dump() is supported for Hash, Array, NilClass,
-TrueClass, FalseClass, Fixnum, Float, Symbol, and String Objects.
+- to_hash() method called if the Object responds to to_hash and the result is converted to JSON
+
+- to_json() method called if the Object responds to to_json
+
+- almost any Object can be dumped, including Exceptions (not including Thread, Mutex and Objects that only make sense within a process)
+
+- default options have been added
 
 ## <a name="description">Description</a>
 
@@ -33,12 +37,117 @@ optimized JSON handling. It was designed as a faster alternative to Yajl and
 other the common Ruby JSON parsers. So far is has achieved that at about 2
 time faster than Yajl for parsing and 3 or more times faster writing JSON.
 
+Oj has several dump or serialization modes which control how Objects are
+converted to JSON. These modes are set with the :effort option in either the
+dafault options or as one of the options to the dump() method. The :strict
+mode will only allow the 7 basic JSON types to be serialized. Any other Object
+will raise and Exception. In the :lazy mode any Object that is not one of the
+JSON types is replaced by a JSON null. In :interal mode any Object will be
+dumped as a JSON Object with keys that match the Ruby Object's variable names
+without the '@' character. This is the highest performance mode. The :internal
+mode is not found in other JSON gems. The last mode, the :tolerant mode is the
+most tolerant. It will serialize any Object but will check to see if the
+Object implements a to_hash() or to_json() method. If either exists that
+method is used for serializing the Object. The to_hash() is more flexible and
+produces more consistent output so it has a preference over the to_json()
+method. If neither the to_json() or to_hash() methods exist then the Oj
+internal Object variable encoding is used.
+
 Coming soon: As an Object marshaller with support for circular references.
 
-Coming soon: A SAX like JSON stream parser.
+Coming soon: A JSON stream parser.
 
 Oj is compatible with Ruby 1.8.7, 1.9.2, 1.9.3, JRuby, and RBX.
 
+## <a name="compare">Comparisons</a>
+
+The following table shows the difference is speeds between several
+serialization packages. The tests had to be scaled back due to limitation of
+some of the gems. I finally gave up trying to get JSON to serialize without
+errors with Ruby 1.9.3. It had internal errors on anything other than a simple
+JSON structure. The errors encountered were:
+
+- MessagePack fails to convert Bignum to JSON
+
+- JSON Pure and Ext fails to serialize any numbers or Objects with the to_json() method
+
+Options were added to the test/perf_simple.rb test to run the test without
+Object encoding and without Bignums. There is also an option for shallow JSON
+so that JSON Pure and Ext can be compared.
+
+None of the packages except Oj were able to serialize Ruby Objects that did
+not have a to_json() method or were of the 7 native JSON types.
+
+It is also worth noting that although Oj is slightly behind MessagePack for
+parsing, Oj serialization is much faster than MessagePack even though Oj uses
+human readable JSON vs the binary MessagePack format.
+
+The results:
+
+with Object and Bignum encoding:
+
+    100000 Oj.load()s in 1.456 seconds or 68.7 loads/msec
+    100000 Yajl::Parser.parse()s in 2.681 seconds or 37.3 parses/msec
+    100000 JSON::Ext::Parser parse()s in 2.804 seconds or 35.7 parses/msec
+    100000 JSON::Pure::Parser parse()s in 27.494 seconds or 3.6 parses/msec
+    MessagePack failed: RangeError: bignum too big to convert into `unsigned long long'
+    100000 Ox.load()s in 3.165 seconds or 31.6 loads/msec
+    Parser results:
+    gem       seconds  parses/msec  X faster than JSON::Pure (higher is better)
+    oj        1.456     68.7        18.9
+    yajl      2.681     37.3        10.3
+    msgpack failed to generate JSON
+    pure     27.494      3.6         1.0
+    ext       2.804     35.7         9.8
+    ox        3.165     31.6         8.7
+    
+    100000 Oj.dump()s in 0.484 seconds or 206.7 dumps/msec
+    100000 Yajl::Encoder.encode()s in 2.167 seconds or 46.2 encodes/msec
+    JSON::Ext failed: TypeError: wrong argument type JSON::Pure::Generator::State (expected Data)
+    JSON::Pure failed: TypeError: wrong argument type JSON::Pure::Generator::State (expected Data)
+    MessagePack failed: RangeError: bignum too big to convert into `unsigned long long'
+    100000 Ox.dump()s in 0.554 seconds or 180.4 dumps/msec
+    Parser results:
+    gem       seconds  dumps/msec  X faster than Yajl (higher is better)
+    oj        0.484    206.7        4.5
+    yajl      2.167     46.2        1.0
+    msgpack failed to generate JSON
+    pure failed to generate JSON
+    ext failed to generate JSON
+    ox        0.554    180.4        3.9
+
+without Objects or numbers (for JSON Pure) JSON:
+
+    100000 Oj.load()s in 0.739 seconds or 135.3 loads/msec
+    100000 Yajl::Parser.parse()s in 1.421 seconds or 70.4 parses/msec
+    100000 JSON::Ext::Parser parse()s in 1.512 seconds or 66.2 parses/msec
+    100000 JSON::Pure::Parser parse()s in 16.953 seconds or 5.9 parses/msec
+    100000 MessagePack.unpack()s in 0.635 seconds or 157.6 packs/msec
+    100000 Ox.load()s in 0.971 seconds or 103.0 loads/msec
+    Parser results:
+    gem       seconds  parses/msec  X faster than JSON::Pure (higher is better)
+    oj        0.739    135.3        22.9
+    yajl      1.421     70.4        11.9
+    msgpack   0.635    157.6        26.7
+    pure     16.953      5.9         1.0
+    ext       1.512     66.2        11.2
+    ox        0.971    103.0        17.5
+    
+    100000 Oj.dump()s in 0.174 seconds or 575.1 dumps/msec
+    100000 Yajl::Encoder.encode()s in 0.729 seconds or 137.2 encodes/msec
+    100000 JSON::Ext generate()s in 7.171 seconds or 13.9 generates/msec
+    100000 JSON::Pure generate()s in 7.219 seconds or 13.9 generates/msec
+    100000 Msgpack()s in 0.299 seconds or 334.8 unpacks/msec
+    100000 Ox.dump()s in 0.210 seconds or 475.8 dumps/msec
+    Parser results:
+    gem       seconds  dumps/msec  X faster than JSON::Pure (higher is better)
+    oj        0.174    575.1       41.5
+    yajl      0.729    137.2        9.9
+    msgpack   0.299    334.8       24.2
+    pure      7.219     13.9        1.0
+    ext       1.512     66.2        4.8
+    ox        0.210    475.8       34.3
+
 ### Simple JSON Writing and Parsing:
 
     require 'oj'

data/ext/oj/dump.c

@@ -34,13 +34,12 @@
 #include <stdio.h>
 #include <string.h>
 
-#include "ruby.h"
-#ifdef HAVE_RUBY_ENCODING_H
+#include "oj.h"
+#if IVAR_HELPERS
 #include "ruby/st.h"
 #else
 #include "st.h"
 #endif
-#include "oj.h"
 
 typedef unsigned long   ulong;
 
@@ -79,13 +78,19 @@ static void	dump_nil(Out out);
 static void	dump_true(Out out);
 static void	dump_false(Out out);
 static void	dump_fixnum(VALUE obj, Out out);
+static void	dump_bignum(VALUE obj, Out out);
 static void	dump_float(VALUE obj, Out out);
 static void	dump_cstr(const char *str, int cnt, Out out);
 static void	dump_hex(u_char c, Out out);
 static void	dump_str(VALUE obj, Out out);
 static void	dump_sym(VALUE obj, Out out);
+static void	dump_class(VALUE obj, Out out);
 static void	dump_array(VALUE obj, int depth, Out out);
 static void	dump_hash(VALUE obj, int depth, Out out);
+static void	dump_data(VALUE obj, Out out);
+static void	dump_object(VALUE obj, int depth, Out out);
+static int	dump_attr_cb(ID key, VALUE value, Out out);
+static void	dump_obj_attrs(VALUE obj, int with_class, int depth, Out out);
 
 static void     grow(Out out, size_t len);
 static int      is_json_friendly(const u_char *str, int len);
@@ -249,6 +254,19 @@ dump_fixnum(VALUE obj, Out out) {
     *out->cur = '\0';
 }
 
+static void
+dump_bignum(VALUE obj, Out out) {
+    VALUE	rs = rb_big2str(obj, 10);
+    int		cnt = (int)RSTRING_LEN(rs);
+
+    if (out->end - out->cur <= (long)cnt) {
+        grow(out, cnt);
+    }
+    memcpy(out->cur, StringValuePtr(rs), cnt);
+    out->cur += cnt;
+    *out->cur = '\0';
+}
+
 static void
 dump_float(VALUE obj, Out out) {
     char	buf[64];
@@ -335,6 +353,38 @@ dump_sym(VALUE obj, Out out) {
     dump_cstr(sym, (int)strlen(sym), out);
 }
 
+static void
+dump_class(VALUE obj, Out out) {
+    switch (out->opts->mode) {
+    case StrictMode:
+        rb_raise(rb_eTypeError, "Failed to dump class %s to JSON.\n", rb_class2name(obj));
+	break;
+    case NullMode:
+	dump_nil(out);
+	break;
+    case ObjectMode:
+    case CompatMode:
+    default:
+	{
+	    const char	*s = rb_class2name(obj);
+	    size_t	len = strlen(s);
+	    size_t	size = len + 20;
+
+	    if (out->end - out->cur <= (long)size) {
+		grow(out, size);
+	    }
+	    memcpy(out->cur, "{\"*\":\"Class\",\"-\":\"", 18);
+	    out->cur += 18;
+	    memcpy(out->cur, s, len);
+	    out->cur += len;
+	    *out->cur++ = '"';
+	    *out->cur++ = '}';
+	    *out->cur = '\0';
+	    break;
+	}
+    }
+}
+
 static void
 dump_array(VALUE a, int depth, Out out) {
     VALUE       *np = RARRAY_PTR(a);
@@ -393,6 +443,9 @@ static void
 dump_hash(VALUE obj, int depth, Out out) {
     int	cnt = (int)RHASH_SIZE(obj);
 
+    if (out->end - out->cur <= 2) {
+	grow(out, 2);
+    }
     *out->cur++ = '{';
     if (0 == cnt) {
 	*out->cur++ = '}';
@@ -411,6 +464,196 @@ dump_hash(VALUE obj, int depth, Out out) {
     *out->cur = '\0';
 }
 
+static void
+dump_data(VALUE obj, Out out) {
+    VALUE   clas = rb_obj_class(obj);
+
+    switch (out->opts->mode) {
+    case StrictMode:
+	rb_raise(rb_eTypeError, "Failed to dump %s Object to JSON in strict mode.\n", rb_class2name(clas));
+	break;
+    case NullMode:
+	dump_nil(out);
+	break;
+    case ObjectMode:
+    case CompatMode:
+    default:
+	if (rb_cTime == clas) {
+	    char        buf[64];
+	    char        *b = buf + sizeof(buf) - 1;
+	    time_t      sec = NUM2LONG(rb_funcall2(obj, oj_tv_sec_id, 0, 0));
+	    long        usec = NUM2LONG(rb_funcall2(obj, oj_tv_usec_id, 0, 0));
+	    char        *dot = b - 7;
+	    long        size;
+
+	    *b-- = '\0';
+	    for (; dot < b; b--, usec /= 10) {
+		*b = '0' + (usec % 10);
+	    }
+	    *b-- = '.';
+	    for (; 0 < sec; b--, sec /= 10) {
+		*b = '0' + (sec % 10);
+	    }
+	    b++;
+	    size = sizeof(buf) - (b - buf) - 1;
+	    if (out->end - out->cur <= size + 20) {
+		grow(out, size + 20);
+	    }
+	    memcpy(out->cur, "{\"*\":\"Time\",\"-\":", 16);
+	    out->cur += 16;
+	    memcpy(out->cur, b, size);
+	    out->cur += size;
+	    *out->cur++ = '}';
+	    *out->cur = '\0';
+	} else {
+	    dump_nil(out);
+	}
+    }
+}
+
+static void
+dump_object(VALUE obj, int depth, Out out) {
+    if (ObjectMode == out->opts->mode) {
+	dump_obj_attrs(obj, 1, depth, out);
+    } else {
+	switch (out->opts->mode) {
+	case StrictMode:
+	    rb_raise(rb_eTypeError, "Failed to dump %s Object to JSON in strict mode.\n", rb_class2name(rb_obj_class(obj)));
+	    break;
+	case NullMode:
+	    dump_nil(out);
+	    break;
+	case ObjectMode:
+	    dump_obj_attrs(obj, 0, depth, out);
+	    break;
+	case CompatMode:
+	default:
+	    if (rb_respond_to(obj, oj_to_hash_id)) {
+	        VALUE	h = rb_funcall(obj, oj_to_hash_id, 0);
+ 
+		if (T_HASH != rb_type(h)) {
+		    rb_raise(rb_eTypeError, "%s.to_hash() did not return a Hash.\n", rb_class2name(rb_obj_class(obj)));
+		}
+		dump_hash(h, depth, out);
+	    } else if (rb_respond_to(obj, oj_to_json_id)) {
+	        VALUE		rs = rb_funcall(obj, oj_to_json_id, 0);
+		const char	*s = StringValuePtr(rs);
+		int		len = (int)RSTRING_LEN(rs);
+
+		if (out->end - out->cur <= len) {
+		    grow(out, len);
+		}
+		memcpy(out->cur, s, len);
+		out->cur += len;
+	    } else {
+		dump_obj_attrs(obj, 0, depth, out);
+	    }
+	    break;
+	}
+    }
+    *out->cur = '\0';
+}
+
+static int
+dump_attr_cb(ID key, VALUE value, Out out) {
+    int		depth = out->depth;
+    size_t	size = depth * out->indent + 1;
+    const char	*attr = rb_id2name(key);
+
+    if (out->end - out->cur <= (long)size) {
+	grow(out, size);
+    }
+    if ('@' == *attr) {
+	attr++;
+    } else {
+	// TBD handle unusual exception mesg data
+    }
+    fill_indent(out, depth);
+    dump_cstr(attr, (int)strlen(attr) - 1, out);
+    *out->cur++ = ':';
+    dump_val(value, depth, out);
+    out->depth = depth;
+    *out->cur++ = ',';
+    
+    return ST_CONTINUE;
+}
+
+static void
+dump_obj_attrs(VALUE obj, int with_class, int depth, Out out) {
+    size_t	size;
+    int		d2 = depth + 1;
+
+    if (out->end - out->cur <= 2) {
+	grow(out, 2);
+    }
+    *out->cur++ = '{';
+    if (with_class) {
+	const char	*class_name = rb_class2name(rb_obj_class(obj));
+	int		clen = (int)strlen(class_name);
+	
+	size = d2 * out->indent + clen + 9;
+	if (out->end - out->cur <= (long)size) {
+	    grow(out, size);
+	}
+	fill_indent(out, d2);
+	*out->cur++ = '"';
+	*out->cur++ = '*';
+	*out->cur++ = '"';
+	*out->cur++ = ':';
+	dump_cstr(class_name, clen, out);
+    }
+    {
+	int	cnt;
+// use encoding as the indicator for Ruby 1.8.7 or 1.9.x
+#if IVAR_HELPERS
+	cnt = (int)rb_ivar_count(obj);
+#else
+	VALUE		vars = rb_funcall2(obj, oj_instance_variables_id, 0, 0);
+	VALUE		*np = RARRAY_PTR(vars);
+	ID		vid;
+	const char	*attr;
+	int		i;
+
+	cnt = (int)RARRAY_LEN(vars);
+#endif
+	if (with_class && 0 < cnt) {
+	    *out->cur++ = ',';
+	}
+	out->depth = depth + 1;
+#if IVAR_HELPERS
+	rb_ivar_foreach(obj, dump_attr_cb, (VALUE)out);
+	out->cur--; // backup to overwrite last comma
+#else
+	size = d2 * out->indent + 1;
+	for (i = cnt; 0 < i; i--, np++) {
+	    if (out->end - out->cur <= (long)size) {
+	        grow(out, size);
+	    }
+	    vid = rb_to_id(*np);
+	    fill_indent(out, d2);
+	    attr = rb_id2name(vid);
+	    if ('@' == *attr) {
+		attr++;
+	    } else {
+		// TBD handle unusual exception mesg data
+	    }
+	    dump_cstr(attr, (int)strlen(attr) - 1, out);
+	    *out->cur++ = ':';
+	    dump_val(rb_ivar_get(obj, vid), d2, out);
+	    if (out->end - out->cur <= 2) {
+	        grow(out, 2);
+	    }
+	    if (1 < i) {
+	      *out->cur++ = ',';
+	    }
+	}
+#endif
+	out->depth = depth;
+    }
+    *out->cur++ = '}';
+    *out->cur = '\0';
+}
+
 static void
 dump_val(VALUE obj, int depth, Out out) {
     switch (rb_type(obj)) {
@@ -419,15 +662,15 @@ dump_val(VALUE obj, int depth, Out out) {
     case T_FALSE:	dump_false(out);		break;
     case T_FIXNUM:	dump_fixnum(obj, out);		break;
     case T_FLOAT:	dump_float(obj, out);		break;
-    case T_BIGNUM:	break; // TBD
+    case T_BIGNUM:	dump_bignum(obj, out);		break;
     case T_STRING:	dump_str(obj, out);		break;
     case T_SYMBOL:	dump_sym(obj, out);		break;
     case T_ARRAY:	dump_array(obj, depth, out);	break;
     case T_HASH:	dump_hash(obj, depth, out);	break;
-    case T_OBJECT:
+    case T_CLASS:	dump_class(obj, out);		break;
+    case T_OBJECT:	dump_object(obj, depth, out);	break;
+    case T_DATA:	dump_data(obj, out);		break;
     case T_REGEXP:
-    case T_CLASS:
-    case T_DATA: // for Time
 	// TBD
 	rb_raise(rb_eNotImpError, "Failed to dump '%s' Object (%02x)\n",
 		 rb_class2name(rb_obj_class(obj)), rb_type(obj));
@@ -460,7 +703,7 @@ dump_obj_to_json(VALUE obj, Options copts, Out out) {
 }
 
 char*
-write_obj_to_str(VALUE obj, Options copts) {
+oj_write_obj_to_str(VALUE obj, Options copts) {
     struct _Out out;
 
     dump_obj_to_json(obj, copts, &out);
@@ -469,7 +712,7 @@ write_obj_to_str(VALUE obj, Options copts) {
 }
 
 void
-write_obj_to_file(VALUE obj, const char *path, Options copts) {
+oj_write_obj_to_file(VALUE obj, const char *path, Options copts) {
     struct _Out out;
     size_t      size;
     FILE        *f;