oj 2.18.5 → 3.16.11

data/README.md

@@ -1,7 +1,13 @@
-# Oj gem
-[![Build Status](https://img.shields.io/travis/ohler55/oj/master.svg)](http://travis-ci.org/ohler55/oj?branch=master) ![Gem](https://img.shields.io/gem/v/oj.svg) ![Gem](https://img.shields.io/gem/dt/oj.svg)
+# [![{}j](http://www.ohler.com/dev/images/oj_comet_64.svg)](http://www.ohler.com/oj) gem
 
-A fast JSON parser and Object marshaller as a Ruby gem.
+[![CI](https://github.com/ohler55/oj/actions/workflows/CI.yml/badge.svg)](https://github.com/ohler55/oj/actions/workflows/CI.yml)
+![Gem](https://img.shields.io/gem/v/oj.svg)
+![Gem](https://img.shields.io/gem/dt/oj.svg)
+[![TideLift](https://tidelift.com/badges/github/ohler55/oj)](https://tidelift.com/subscription/pkg/rubygems-oj?utm_source=rubygems-oj&utm_medium=referral&utm_campaign=readme)
+
+A *fast* JSON parser and Object marshaller as a Ruby gem.
+
+Version 3.13 is out with a much faster parser (`Oj::Parser`) and option isolation.
 
 ## Using
 
@@ -25,11 +31,6 @@ puts "Same? #{h == h2}"
 # true
 ```
 
-By default Oj uses the :object mode which is used to marshal and unmarshal Ruby
-objects. Deserialize arbitrary JSON in object mode may lead to unexpected
-results. :compat mode is a better choice for rails and :strict mode is a better
-choice for general JSON parsing. See the options section below for details.
-
 ## Installation
 ```
 gem install oj
@@ -41,247 +42,78 @@ or in Bundler:
 gem 'oj'
 ```
 
-## Advanced
-
-Optimized JSON (Oj), as the name implies, was written to provide speed optimized
-JSON handling. It was designed as a faster alternative to Yajl and other
-common Ruby JSON parsers. So far it has achieved that, and is about 2 times faster
-than any other Ruby JSON parser, and 3 or more times faster at serializing JSON.
-
-Oj has several `dump` or serialization modes which control how Ruby `Object`s are
-converted to JSON. These modes are set with the `:mode` option in either the
-default options or as one of the options to the `dump` method. In addition to
-the various options there are also alternative APIs for parsing JSON.
-
-The fastest alternative parser API is the `Oj::Doc` API. The `Oj::Doc` API takes
-a completely different approach by opening a JSON document and providing calls
-to navigate around the JSON while it is open. With this approach, JSON access
-can be well over 20 times faster than conventional JSON parsing.
-
-The `Oj::Saj` and `Oj::ScHandler` APIs are callback parsers that
-walk the JSON document depth first and makes callbacks for each element.
-Both callback parser are useful when only portions of the JSON are of
-interest. Performance up to 20 times faster than conventional JSON is
-possible if only a few elements of the JSON are of interest.
-
-### Options
-
-To change default serialization mode use the following form. Attempting to
-modify the Oj.default_options Hash directly will not set the changes on the
-actual default options but on a copy of the Hash:
-
-```ruby
-Oj.default_options = {:mode => :compat }
-```
-
-### Common (for serializer and parser) options
-
- * `:mode` [Symbol] mode for dumping and loading JSON. **Important format details [here](http://www.ohler.com/dev/oj_misc/encoding_format.html)**
-
-    - `:object` mode will dump any `Object` as a JSON `Object` with keys that
-      match the Ruby `Object`'s variable names without the '@' prefix
-      character. This mode has the best performance and is the default
-
-    - `:strict` mode will only allow the 7 basic JSON types to be serialized. Any
-      other `Object` will raise an `Exception`
-
-    - `:null` mode replaces any `Object` that is not one of the JSON types with a JSON `null`
-
-    - `:compat` mode attempts to be compatible with other systems. It will
-      serialize any `Object`, but will check to see if the `Object` implements an
-      `to_hash` or `to_json` method. If either exists, that method is used for
-      serializing the `Object`.  Since `as_json` is more flexible and produces
-      more consistent output, it is preferred over the `to_json` method. If
-      neither the `to_json` or `to_hash` methods exists, then the Oj internal
-      `Object` variable encoding is used
-
- * `:time_format` [Symbol] time format when dumping in :compat and :object mode
-
-    - `:unix` time is output as a decimal number in seconds since epoch including
-      fractions of a second.
-
-    - `:unix_zone` similar to the `:unix` format but with the timezone encoded in
-      the exponent of the decimal number of seconds since epoch.
-
-    - `:xmlschema` time is output as a string that follows the XML schema definition.
-
-    - `:ruby` time is output as a string formatted using the Ruby `to_s` conversion.
-
- * `:second_precision` [Fixnum] number of digits after the decimal when dumping
-   the seconds portion of time
-
- * `:bigdecimal_as_decimal` [Boolean] dump BigDecimal as a decimal number
-   or as a String
-
- * `:create_id` [String] create id for json compatible object encoding,
-   default is `json_create`.
-
- * `:allow_gc` [Boolean] allow or prohibit GC during parsing, default is
-   true (allow).
-
- * `:quirks_mode` [Boolean] Allow single JSON values instead of
-   documents, default is true (allow).
-
- * `:allow_invalid_unicode` [Boolean] Allow invalid unicode, default is
-   false (don't allow)
-
- * `:omit_nil` [Boolean] If true, Hash and Object attributes with nil values
-   are omitted
-
-### Serializer options
-
- * `:indent` [Fixnum] number of spaces to indent each element in an JSON
-   document, zero is no newline between JSON elements, negative indicates no
-   newline between top level JSON elements in a stream
-   
- * `:circular` [Boolean] support circular references while dumping
-
- * `:escape_mode` [Symbol] determines the characters to escape
-
-    - `:newline` allows unescaped newlines in the output
+## Rails and json quickstart
 
-    - `:json` follows the JSON specification. This is the default mode
+See the Quickstart sections of the [Rails](pages/Rails.md) and [json](pages/JsonGem.md) docs.
 
-    - `:xss_safe` escapes HTML and XML characters such as `&` and `<`
+## multi_json
 
-    - `:ascii` escapes all non-ascii or characters with the hi-bit set
+Code which uses [multi_json](https://github.com/intridea/multi_json)
+will automatically prefer Oj if it is installed.
 
- * `:use_to_json` [Boolean] call `to_json()` methods on dump, default is
-   false
+## Support
 
- * `:use_as_json` [Boolean] call `as_json()` methods on dump, default is
-   false
+[Get supported Oj with a Tidelift Subscription.](https://tidelift.com/subscription/pkg/rubygems-oj?utm_source=rubygems-oj&utm_medium=referral&utm_campaign=readme) Security updates are [supported](https://tidelift.com/security).
 
- * `:nan` [Symbol] How to dump Infinity, -Infinity, and
-   NaN in :null, :strict, and :compat mode. Default is :auto
+## Further Reading
 
-    - `:null` places a null
+For more details on options, modes, advanced features, and more follow these
+links.
 
-    - `:huge` places a huge number
-
-    - `:word` places Infinity or NaN
-
-    - `:raise` raises and exception
-
-    - `:auto` uses default for each mode which are `:raise` for `:strict`, `:null` for `:null`, and `:word` for `:compat`
-
-### Parser options
-
- * `:auto_define` [Boolean] automatically define classes if they do not
-   exist
-
- * `:symbol_keys` [Boolean] use symbols instead of strings for hash keys
-
- * `:class_cache` [Boolean] cache classes for faster parsing (if
-   dynamically modifying classes or reloading classes then don't use this)
-
- * `:bigdecimal_load` [Symbol] load decimals as BigDecimal instead of as a
-   Float
-
-    - `:bigdecimal` convert all decimal numbers to BigDecimal
-
-    - `:float` convert all decimal numbers to Float
-
-    - `:auto` the most precise for the number of digits is used
-
- * `:float_precision` [Fixnum] number of digits of precision when dumping
-   floats, 0 indicates use Ruby
-
- * `:nilnil` [Boolean] if true a nil input to load will return nil and
-   not raise an Exception
-
- * `:empty_string` [Boolean] if true an empty input will not raise an
-   Exception, default is true (allow). When Oj.mimic_JSON is used,
-   default is false (raise exception when empty string is encountered)
-
- * `:hash_class` [Class] Class to use instead of Hash on load
+ - [{file:Options.md}](pages/Options.md) for parse and dump options.
+ - [{file:Modes.md}](pages/Modes.md) for details on modes for strict JSON compliance, mimicking the JSON gem, and mimicking Rails and ActiveSupport behavior.
+ - [{file:JsonGem.md}](pages/JsonGem.md) includes more details on json gem compatibility and use.
+ - [{file:Rails.md}](pages/Rails.md) includes more details on Rails and ActiveSupport compatibility and use.
+ - [{file:Custom.md}](pages/Custom.md) includes more details on Custom mode.
+ - [{file:Encoding.md}](pages/Encoding.md) describes the :object encoding format.
+ - [{file:Compatibility.md}](pages/Compatibility.md) lists current compatibility with Rubys and Rails.
+ - [{file:Advanced.md}](pages/Advanced.md) for fast parser and marshalling features.
+ - [{file:Security.md}](pages/Security.md) for security considerations.
+ - [{file:InstallOptions.md}](pages/InstallOptions.md) for install option.
 
 ## Releases
 
-See [CHANGELOG.md](CHANGELOG.md)
-
-## Compatibility
-
-**Ruby**
-
-Oj is compatible with Ruby 1.9.3, 2.0.0, 2.1, 2.2, 2.3, 2.4 and RBX.
-Support for JRuby has been removed as JRuby no longer supports C extensions and
-there are bugs in the older versions that are not being fixed.
-
-**Rails**
-
-Although up until 4.1 Rails uses [multi_json](https://github.com/intridea/multi_json), an [issue in Rails](https://github.com/rails/rails/issues/9212) causes ActiveSupport to fail to make use Oj for JSON handling.
-There is a
-[gem to patch this](https://github.com/GoodLife/rails-patch-json-encode) for
-Rails 3.2 and 4.0. As of the Oj 2.6.0 release the default behavior is to not use
-the `to_json()` method unless the `:use_to_json` option is set. This provides
-another work around to the rails older and newer behavior.
-
-The latest ActiveRecord is able to work with Oj by simply using the line:
-
-```
-serialize :metadata, Oj
-```
-
-In version Rails 4.1, multi_json has been removed, and this patch is unnecessary and will no longer work.
-Instead, use the `oj_mimic_json` [gem](https://github.com/ohler55/oj_mimic_json) along with `oj` in your `Gemfile` to have Oj mimic the JSON gem and be used in its place by `ActiveSupport` JSON handling:
-
-```
-gem 'oj'
-gem 'oj_mimic_json'
-```
-
-## Security and Optimization
-
-Two settings in Oj are useful for parsing but do expose a vulnerability if used
-from an untrusted source. Symbolized keys can cause memory to be filled with
-previous versions of ruby. Ruby 2.1 and below does not garbage collect
-Symbols. The same is true for auto defining classes in all versions of ruby;
-memory will also be exhausted if too many classes are automatically
-defined. Auto defining is a useful feature during development and from trusted
-sources but it allows too many classes to be created in the object load mode and
-auto defined is used with an untrusted source. The `Oj.strict_load()` method
-sets and uses the most strict and safest options. It should be used by
-developers who find it difficult to understand the options available in Oj.
-
-The options in Oj are designed to provide flexibility to the developer. This
-flexibility allows Objects to be serialized and deserialized. No methods are
-ever called on these created Objects but that does not stop the developer from
-calling methods on them. As in any system, check your inputs before working with
-them. Taking an arbitrary `String` from a user and evaluating it is never a good
-idea from an unsecure source. The same is true for `Object` attributes as they
-are not more than `String`s. Always check inputs from untrusted sources.
+See [{file:CHANGELOG.md}](CHANGELOG.md) and [{file:RELEASE_NOTES.md}](RELEASE_NOTES.md)
 
 ## Links
 
-*Documentation*: http://www.ohler.com/oj, http://rubydoc.info/gems/oj
+- *Documentation*: http://www.ohler.com/oj/doc, http://rubydoc.info/gems/oj
 
-*GitHub* *repo*: https://github.com/ohler55/oj
+- *GitHub* *repo*: https://github.com/ohler55/oj
 
-*RubyGems* *repo*: https://rubygems.org/gems/oj
+- *RubyGems* *repo*: https://rubygems.org/gems/oj
 
-Follow [@peterohler on Twitter](http://twitter.com/#!/peterohler) for announcements and news about the Oj gem.
+Follow [@peterohler on Twitter](http://twitter.com/peterohler) for announcements and news about the Oj gem.
 
 #### Performance Comparisons
 
-[Oj Strict Mode Performance](http://www.ohler.com/dev/oj_misc/performance_strict.html) compares Oj strict mode parser performance to other JSON parsers.
+ - [Oj Strict Mode Performance](http://www.ohler.com/dev/oj_misc/performance_strict.html) compares Oj strict mode parser performance to other JSON parsers.
 
-[Oj Compat Mode Performance](http://www.ohler.com/dev/oj_misc/performance_compat.html) compares Oj compat mode parser performance to other JSON parsers.
+ - [Oj Compat Mode Performance](http://www.ohler.com/dev/oj_misc/performance_compat.html) compares Oj compat mode parser performance to other JSON parsers.
 
-[Oj Object Mode Performance](http://www.ohler.com/dev/oj_misc/performance_object.html) compares Oj object mode parser performance to other marshallers.
+ - [Oj Object Mode Performance](http://www.ohler.com/dev/oj_misc/performance_object.html) compares Oj object mode parser performance to other marshallers.
 
-[Oj Callback Performance](http://www.ohler.com/dev/oj_misc/performance_callback.html) compares Oj callback parser performance to other JSON parsers.
+ - [Oj Callback Performance](http://www.ohler.com/dev/oj_misc/performance_callback.html) compares Oj callback parser performance to other JSON parsers.
 
 #### Links of Interest
 
-*Fast XML parser and marshaller on RubyGems*: https://rubygems.org/gems/ox
+ - *Fast XML parser and marshaller on RubyGems*: https://rubygems.org/gems/ox
+
+ - *Fast XML parser and marshaller on GitHub*: https://github.com/ohler55/ox
+
+ - [Need for Speed](http://www.ohler.com/dev/need_for_speed/need_for_speed.html) for an overview of how Oj::Doc was designed.
+
+ - *OjC, a C JSON parser*: https://www.ohler.com/ojc also at https://github.com/ohler55/ojc
 
-*Fast XML parser and marshaller on GitHub*: https://github.com/ohler55/ox
+ - *Agoo, a high performance Ruby web server supporting GraphQL on GitHub*: https://github.com/ohler55/agoo
 
-[Oj Object Encoding Format](http://www.ohler.com/dev/oj_misc/encoding_format.html) describes the OJ Object JSON encoding format.
+ - *Agoo-C, a high performance C web server supporting GraphQL on GitHub*: https://github.com/ohler55/agoo-c
 
-[Need for Speed](http://www.ohler.com/dev/need_for_speed/need_for_speed.html) for an overview of how Oj::Doc was designed.
+ - *oj-introspect, an example of creating an Oj parser extension in C*: https://github.com/meinac/oj-introspect
 
-*OjC, a C JSON parser*: https://www.ohler.com/ojc also at https://github.com/ohler55/ojc
+#### Contributing
 
-*Piper Push Cache, push JSON to browsers*: http://www.piperpushcache.com
++ Provide a Pull Request off the `develop` branch.
++ Report a bug
++ Suggest an idea
++ Code is now formatted with the clang-format tool with the configuration file in the root of the repo.

data/RELEASE_NOTES.md

@@ -0,0 +1,61 @@
+# RELEASE NOTES
+
+The release notes here are organized by release. For a list of changes
+see the See [{file:CHANGELOG.md}](CHANGELOG.md) file. In this file are
+the steps to take to aid in keeping things rolling after updating to
+the latest version.
+
+## 3.13.7
+
+The default for JSON when mimicked by Oj is now to set
+`:allow_invalid_unicode`. To change that behavior JSON.load, set that
+option to false.
+
+## 3.13.x
+
+This release included a new cache that performs better than the
+earlier cache and a new high performance parser.
+
+### Cache
+
+The new cache includes a least recently used expiration to reduce
+memory use. The cache is also self adjusting and will expand as needed
+for better performance. It also handles Hash keys and string values
+with two options, `:cache_keys`, a boolean and `:cache_str` an
+integer. The `:cache_str` if set to more than zero is the limit for
+the length of string values to cache. The maximum value is 35 which
+allows strings up to 34 bytes to be cached.
+
+One interesting aspect of the cache is not so much the string caching
+which performs similar to the Ruby intern functions but the caching of
+symbols and object attribute names. There is a significant gain for
+symbols and object attributes.
+
+If the cache is not desired then setting the default options to turn
+it off can be done with this line:
+
+``` ruby
+Oj.default_options = { cache_keys: false, cache_str: 0 }
+```
+
+### Oj::Parser
+
+The new parser uses a different core that follows the approach taken
+by [OjC](https://github.com/ohler55/ojc) and
+[OjG](https://github.com/ohler55/ojg). It also takes advantage of the
+bulk Array and Hash functions. Another issue the new parser addresses
+is option management. Instead of a single global default_options each
+parser instance maintains it's own options.
+
+There is a price to be paid when using the Oj::Parser. The API is not
+the same the older parser. A single parser can only be used in a
+single thread. This allows reuse of internal buffers for additional
+improvements in performance.
+
+The performane advantage of the Oj::Parse is that it is more than 3
+times faster than the Oj::compat_load call and 6 times faster than the
+JSON gem.
+
+### Dump Performance
+
+Thanks to Watson1978 Oj.dump also received a speed boost.

data/ext/oj/buf.h

@@ -1,103 +1,85 @@
-/* buf.h
- * Copyright (c) 2011, Peter Ohler
- * All rights reserved.
- * 
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- * 
- *  - Redistributions of source code must retain the above copyright notice, this
- *    list of conditions and the following disclaimer.
- * 
- *  - Redistributions in binary form must reproduce the above copyright notice,
- *    this list of conditions and the following disclaimer in the documentation
- *    and/or other materials provided with the distribution.
- * 
- *  - Neither the name of Peter Ohler nor the names of its contributors may be
- *    used to endorse or promote products derived from this software without
- *    specific prior written permission.
- * 
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
- * DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
- * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
- * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
- * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
+// Copyright (c) 2011 Peter Ohler. All rights reserved.
+// Licensed under the MIT License. See LICENSE file in the project root for license details.
 
-#ifndef __OJ_BUF_H__
-#define __OJ_BUF_H__
+#ifndef OJ_BUF_H
+#define OJ_BUF_H
 
+#include "mem.h"
 #include "ruby.h"
 
-typedef struct _Buf {
-    char	*head;
-    char	*end;
-    char	*tail;
-    char	base[1024];
+typedef struct _buf {
+    char *head;
+    char *end;
+    char *tail;
+    char  base[1024];
 } *Buf;
 
-inline static void
-buf_init(Buf buf) {
+inline static void buf_init(Buf buf) {
     buf->head = buf->base;
-    buf->end = buf->base + sizeof(buf->base) - 1;
+    buf->end  = buf->base + sizeof(buf->base) - 1;
     buf->tail = buf->head;
 }
 
-inline static void
-buf_cleanup(Buf buf) {
+inline static void buf_reset(Buf buf) {
+    buf->tail = buf->head;
+}
+
+inline static void buf_cleanup(Buf buf) {
     if (buf->base != buf->head) {
-        xfree(buf->head);
+        OJ_R_FREE(buf->head);
     }
 }
 
-inline static size_t
-buf_len(Buf buf) {
+inline static size_t buf_len(Buf buf) {
     return buf->tail - buf->head;
 }
 
-inline static void
-buf_append_string(Buf buf, const char *s, size_t slen) {
+inline static const char *buf_str(Buf buf) {
+    *buf->tail = '\0';
+    return buf->head;
+}
+
+inline static void buf_append_string(Buf buf, const char *s, size_t slen) {
+    if (0 == slen) {
+        return;
+    }
+
     if (buf->end <= buf->tail + slen) {
-	size_t	len = buf->end - buf->head;
-	size_t	toff = buf->tail - buf->head;
-	size_t	new_len = len + slen + len / 2;
+        size_t len     = buf->end - buf->head;
+        size_t toff    = buf->tail - buf->head;
+        size_t new_len = len + slen + len / 2;
 
-	if (buf->base == buf->head) {
-	    buf->head = ALLOC_N(char, new_len);
-	    memcpy(buf->head, buf->base, len);
-	} else {
-	    REALLOC_N(buf->head, char, new_len);
-	}
-	buf->tail = buf->head + toff;
-	buf->end = buf->head + new_len - 1;
+        if (buf->base == buf->head) {
+            buf->head = OJ_R_ALLOC_N(char, new_len);
+            memcpy(buf->head, buf->base, len);
+        } else {
+            OJ_R_REALLOC_N(buf->head, char, new_len);
+        }
+        buf->tail = buf->head + toff;
+        buf->end  = buf->head + new_len - 1;
     }
     memcpy(buf->tail, s, slen);
     buf->tail += slen;
 }
-    
-inline static void
-buf_append(Buf buf, char c) {
+
+inline static void buf_append(Buf buf, char c) {
     if (buf->end <= buf->tail) {
-	size_t	len = buf->end - buf->head;
-	size_t	toff = buf->tail - buf->head;
-	size_t	new_len = len + len / 2;
+        size_t len     = buf->end - buf->head;
+        size_t toff    = buf->tail - buf->head;
+        size_t new_len = len + len / 2;
 
-	if (buf->base == buf->head) {
-	    buf->head = ALLOC_N(char, new_len);
-	    memcpy(buf->head, buf->base, len);
-	} else {
-	    REALLOC_N(buf->head, char, new_len);
-	}
-	buf->tail = buf->head + toff;
-	buf->end = buf->head + new_len - 1;
+        if (buf->base == buf->head) {
+            buf->head = OJ_R_ALLOC_N(char, new_len);
+            memcpy(buf->head, buf->base, len);
+        } else {
+            OJ_R_REALLOC_N(buf->head, char, new_len);
+        }
+        buf->tail = buf->head + toff;
+        buf->end  = buf->head + new_len - 1;
     }
     *buf->tail = c;
     buf->tail++;
     //*buf->tail = '\0'; // for debugging
 }
 
-#endif /* __OJ_BUF_H__ */
+#endif /* OJ_BUF_H */