oj 3.11.5 → 3.16.5

data/pages/Options.md

@@ -66,17 +66,35 @@ Determines how to load decimals.
 
  - `:auto` the most precise for the number of digits is used.
 
+ - `:fast` faster conversion to Float.
+
+ - `:ruby` convert to Float using the Ruby `to_f` conversion.
+
 This can also be set with `:decimal_class` when used as a load or
 parse option to match the JSON gem. In that case either `Float`,
 `BigDecimal`, or `nil` can be provided.
 
-### :compat_bigdecimal [Boolean]
+### :cache_keys [Boolean]
 
-Determines how to load decimals when in `:compat` mode.
+If true Hash keys are cached or interned. There are trade-offs with
+caching keys. Large caches will use more memory and in extreme cases
+(like over a million) the cache may be slower than not using
+it. Repeated parsing of similar JSON docs is where cache_keys shines
+especially with symbol keys.
 
- - `true` convert all decimal numbers to BigDecimal.
+There is a maximum length for cached keys. Any key longer than 34
+bytes is not cached. Everything still works but the key is not cached.
 
- - `false` convert all decimal numbers to Float.
+### :cache_strings [Int]
+
+Shorter strings can be cached for better performance. A limit,
+cache_strings, defines the upper limit on what strings are cached. As
+with cached keys only strings less than 35 bytes are cached even if
+the limit is set higher. Setting the limit to zero effectively
+disables the caching of string values.
+
+Note that caching for strings is for string values and not Hash keys
+or Object attributes.
 
 ### :circular [Boolean]
 
@@ -90,6 +108,14 @@ recreate the looped references on load.
 Cache classes for faster parsing. This option should not be used if
 dynamically modifying classes or reloading classes then don't use this.
 
+### :compat_bigdecimal [Boolean]
+
+Determines how to load decimals when in `:compat` mode.
+
+ - `true` convert all decimal numbers to BigDecimal.
+
+ - `false` convert all decimal numbers to Float.
+
 ### :create_additions
 
 A flag indicating that the :create_id key, when encountered during parsing,
@@ -132,6 +158,8 @@ Determines the characters to escape when dumping. Only the :ascii and
 
  - `:json` follows the JSON specification. This is the default mode.
 
+ - `:slash` escapes `/` characters.
+
  - `:xss_safe` escapes HTML and XML characters such as `&` and `<`.
 
  - `:ascii` escapes all non-ascii or characters with the hi-bit set.
@@ -237,6 +265,10 @@ to true.
 
 The number of digits after the decimal when dumping the seconds of time.
 
+### :skip_null_byte [Boolean]
+
+If true, null bytes in strings will be omitted when dumping.
+
 ### :space
 
 String inserted after the ':' character when dumping a JSON object. The
@@ -251,7 +283,13 @@ compatibility. Using just indent as an integer gives better performance.
 
 ### :symbol_keys [Boolean]
 
-Use symbols instead of strings for hash keys. :symbolize_names is an alias.
+Use symbols instead of strings for hash keys.
+
+### :symbolize_names [Boolean]
+
+Like :symbol_keys has keys are made into symbols but only when
+mimicking the JSON gem and then only as the JSON gem honors it so
+JSON.parse honors the option but JSON.load does not.
 
 ### :trace
 

data/pages/Parser.md

@@ -0,0 +1,309 @@
+# How Oj Just Got Faster
+
+The original Oj parser is a performant parser that supports several
+modes. As of this writing Oj is almost 10 years old. A dinosaur by
+coding standards. It was time for an upgrade. Dealing with issues over
+the years it became clear that a few things could have been done
+better. The new `Oj::Parser` is a response that not only attempts to
+address some of the issues but also give the Oj parser a significant
+boost in performance. `Oj::Parser` takes a different approach to JSON
+parsing than the now legacy Oj parser. Not really a legacy parser yet
+since the `Oj::Parser` is not a drop-in replacement for the JSON gem
+but it is as much 3 times or more faster than the previous parser in
+some modes.
+
+## Address Issues
+
+There are a few features of the`Oj.load` parser that continue to be
+the reason for many of the issue on the project. The most significant
+area is compatibility with both Rails and the JSON gem as they battle
+it out for which behavior will win out in any particular
+situation. Most of the issues are on the writing or dumping side of
+the JSON packages but some are present on the parsing as
+well. Conversion of decimals is one area where the Rails and the JSON
+gem vary. The `Oj::Parser` addresses this by allowing for completely
+separate parser instances. Create a parser and configure it for the
+situation and leave the others parsers on their own.
+
+The `Oj::Parser` is mostly compatible with the JSON gem and Rails but
+no claims are made that the behavior will be the same as either.
+
+The most frequent issues that can addressed with the new parser are
+around the handling of options. For `Oj.load` there is a set of
+default options that can be set and the same options can be specified
+for each call to parse or load. This approach as a couple of
+downsides. One the defaults are shared across all calls to parse no
+matter what the desire mode is. The second is that having to provide
+all the options on each parse call incurs a performance penalty and is
+just annoying to repeat the same set of options over may calls.
+
+By localizing options to a specific parser instance there is never any
+bleed over to other instances.
+
+## How
+
+It's wonderful to wish for a faster parser that solves all the
+annoyances of the previous parser but how was it done is a much more
+interesting question to answer.
+
+At the core, the API for parsing was changed. Instead of a sinle
+global parser any number of parsers can be created and each is separate
+from the others. The parser itself is able to rip through a JSON
+string, stream, or file and then make calls to a delegate to process
+the JSON elements according to the delegate behavior. This is similar
+to the `Oj.load` parser but the new parser takes advantage of
+character maps, reduced conditional branching, and calling function
+pointers.
+
+### Options
+
+As mentioned, one way to change the options issues was to change the
+API. Instead of having a shared set of default options a separate
+parser is created and configured for each use case. Options are set
+with methods on the parser so no more guessing what options are
+available. With options isolated to individual parsers there is no
+unintended leakage to other parse use cases.
+
+### Structure
+
+A relative small amount of time is spent in the actual parsing of JSON
+in `Oj.load`. Most of the time is spent building the Ruby
+Objects. Even cutting the parsing time in half only gives a 10%
+improvement in performance but 10% is still an improvement.
+
+The `Oj::Parser` is designed to reduce conditional branching. To do
+that it uses character maps for the various states that the parser
+goes through when parsing. There is no recursion as the JSON elements
+are parsed. The use of a character maps for each parser state means
+the parser function can and is re-entrant so partial blocks of JSON
+can be parsed and the results combined.
+
+There are no Ruby calls in the parser itself. Instead delegates are
+used to implement the various behaviors of the parser which are
+currently validation (validate), callbacks (SAJ), or building Ruby
+objects (usual). The delegates are where all the Ruby calls and
+related optimizations take place.
+
+Considering JSON file parsing, `Oj.load_file` is able to read a file a
+block at a time and the new `Oj::Parser` does the same. There was a
+change in how that is done though. `Oj.load_file` sets up a reader
+that must be called for each character. Basically a buffered
+reader. `Oj::Parser` drops down a level and uses a re-entrant parser
+that takes a block of bytes at a time so there is no call needed for
+each character but rather just iterating over the block read from the
+file.
+
+Reading a block at a time also allows for an efficient second thread
+to be used for reading blocks. That feature is not in the first
+iteration of the `Oj::Parser` but the stage is set for it in the
+future. The same approach was used successfully in
+[OjC](https://github.com/ohler55/ojc) which is where the code for the
+parser was taken from.
+
+### Delegates
+
+There are three delegates; validate, SAJ, and usual.
+
+#### Validate
+
+The validate delegate is trivial in that does nothing other than let
+the parser complete. There are no options for the validate
+delegate. By not making any Ruby calls other than to start the parsing
+the validate delegate is no surprise that the validate delegate is the
+best performer.
+
+#### SAJ (Simple API for JSON)
+
+The SAJ delegate is compatible with the SAJ handlers used with
+`Oj.saj_parse` so it needs to keep track of keys for the
+callbacks. Two optimizations are used. The first is a reuseable key
+stack while the second is a string cache similar to the Ruby intern
+function.
+
+When parsing a Hash (JSON object) element the key is passed to the
+callback function if the SAJ handler responds to the method. The key
+is also provided when closing an Array or Hash that is part of a
+parent Hash. A key stack supports this.
+
+If the option is turned on a lookup is made and previously cached key
+VALUEs are used. This avoids creating the string for the key and
+setting the encoding on it. The cache used is a auto expanding hash
+implementation that is limited to strings less than 35 characters
+which covers most keys. Larger strings use the slower string creation
+approach. The use of the cache reduces object creation which save on
+both memory allocation and time. It is not appropriate for one time
+parsing of say all the keys in a dictionary but is ideally suited for
+loading similar JSON multiple times.
+
+#### Usual
+
+By far the more complex of the delegates is the 'usual' delegate. The
+usual delegate builds Ruby Objects when parsing JSON. It incorporates
+many options for configuration and makes use of a number of
+optimizations.
+
+##### Reduce Branching
+
+In keeping with the goal of reducing conditional branching most of the
+delegate options are implemented by changing a function pointer
+according to the option selected. For example when turning on or off
+`:symbol_keys` the function to calculate the key is changed so no
+decision needs to be made during parsing. Using this approach option
+branching happens when the option is set and not each time when
+parsing.
+
+##### Cache
+
+Creating Ruby Objects whether Strings, Array, or some other class is
+expensive. Well expensive when running at the speeds Oj runs at. One
+way to reduce Object creation is to cache those objects on the
+assumption that they will most likely be used again. This is
+especially true of Hash keys and Object attribute IDs. When creating
+Objects from a class name in the JSON a class cache saves resolving
+the string to a class each time. Of course there are times when
+caching is not preferred so caching can be turned on or off with
+option methods on the parser which are passed down to the delegate..
+
+The Oj cache implementation is an auto expanding hash. When certain
+limits are reached the hash is expanded and rehashed. Rehashing can
+take some time as the number of items cached increases so there is
+also an option to start with a larger cache size to avoid or reduce
+the likelihood of a rehash.
+
+The Oj cache has an advantage over the Ruby intern function
+(`rb_intern()`) in that several steps are needed for some cached
+items. As an example Object attribute IDs are created by adding an `@`
+character prefix to a string and then converting to a ID. This is done
+once when inserting into the cache and after that only a lookup is
+needed.
+
+##### Bulk Insert
+
+The Ruby functions available for C extension functions are extensive
+and offer many options across the board. The bulk insert functions for
+both Arrays and Hashes are much faster than appending or setting
+functions that set one value at a time. The Array bulk insert is
+around 15 times faster and for Hash it is about 3 times faster.
+
+To take advantage of the bulk inserts arrays of VALUEs are
+needed. With a little planning there VALUE arrays can be reused which
+leads into another optimization, the use of stacks.
+
+##### Stacks
+
+Parsing requires memory to keep track of values when parsing nested
+JSON elements. That can be done on the call stack making use of
+recursive calls or it can be done with a stack managed by the
+parser. The `Oj.load` method maintains a stack for Ruby object and
+builds the output as the parsing progresses.
+
+`Oj::Parser` uses three different stacks. One stack for values, one
+for keys, and one for collections (Array and Hash). By postponing the
+creation of the collection elements the bulk insertions for Array and
+Hash can be used. For arrays the use of a value stack and creating the
+array after all elements have been identified gives a 15x improvement
+in array creation.
+
+For Hash the story is a little different. The bulk insert for Hash
+alternates keys and values but there is a wrinkle to consider. Since
+Ruby Object creation is triggered by the occurrence of an element that
+matches a creation identifier the creation of a collection is not just
+for Array and Hash but also Object. Setting Object attributes uses an
+ID and not a VALUE. For that reason the keys should not be created as
+String or Symbol types as they would be ignored and the VALUE creation
+wasted when setting Object attributes. Using the bulk insert for Hash
+gives a 3x improvement for that part of the object building.
+
+Looking at the Object creation the JSON gem expects a class method of
+`#json_create(arg)`. The single argument is the Hash resulting from
+the parsing assuming that the parser parsed to a Hash first. This is
+less than ideal from a performance perspective so `Oj::Parser`
+provides an option to take that approach or to use the much more
+efficient approach of never creating the Hash but instead creating the
+Object and then setting the attributes directly.
+
+To further improve performance and reduce the amount of memory
+allocations and frees the stacks are reused from one call to `#parse`
+to another.
+
+## Results
+
+The results are even better than expected. Running the
+[perf_parser.rb](https://github.com/ohler55/oj/blob/develop/test/perf_parser.rb)
+file shows the improvements. There are four comparisons all run on a
+MacBook Pro with Intel processor.
+
+### Validation
+
+Without a comparible parser that just validates a JSON document the
+`Oj.saj_parse` callback parser with a nil handler is used for
+comparison to the new `Oj::Parser.new(:validate)`. In that case the
+comparison is:
+
+```
+             System  time (secs)  rate (ops/sec)
+-------------------  -----------  --------------
+Oj::Parser.validate       0.101      494369.136
+       Oj::Saj.none       0.205      244122.745
+```
+
+The `Oj::Parser.new(:validate)` is **2.03** times faster!
+
+### Callback
+
+Oj has two callback parsers. One is SCP and the other SAJ. Both are
+similar in that a handler is provided that implements methods for
+processing the various element types in a JSON document. Comparing
+`Oj.saj_parse` to `Oj::Parser.new(:saj)` with a all callback methods
+implemented handler gives the following raw results:
+
+```
+        System  time (secs)  rate (ops/sec)
+--------------  -----------  --------------
+Oj::Parser.saj       0.783       63836.986
+   Oj::Saj.all       1.182       42315.397
+```
+
+The `Oj::Parser.new(:saj)` is **1.51** times faster.
+
+### Parse to Ruby primitives
+
+Parsing to Ruby primitives and Array and Hash is possible with most
+parsers including the JSON gem parser. The raw results comparing
+`Oj.strict_load`, `Oj::Parser.new(:usual)`, and the JSON gem are:
+
+```
+          System  time (secs)  rate (ops/sec)
+----------------  -----------  --------------
+Oj::Parser.usual       0.452      110544.876
+ Oj::strict_load       0.699       71490.257
+       JSON::Ext       1.009       49555.094
+```
+
+The `Oj::Parser.new(:saj)` is **1.55** times faster than `Oj.load` and
+**2.23** times faster than the JSON gem.
+
+### Object
+
+Oj supports two modes for Object serialization and
+deserialization. Comparing to the JSON gem compatible mode
+`Oj.compat_load`, `Oj::Parser.new(:usual)`, and the JSON gem yields
+the following raw results:
+
+```
+          System  time (secs)  rate (ops/sec)
+----------------  -----------  --------------
+Oj::Parser.usual       0.071      703502.033
+ Oj::compat_load       0.225      221762.927
+       JSON::Ext       0.401      124638.859
+```
+
+The `Oj::Parser.new(:saj)` is **3.17** times faster than
+`Oj.compat_load` and **5.64** times faster than the JSON gem.
+
+## Summary
+
+With a performance boost of from 1.5x to over 3x over the `Oj.load`
+parser the new `Oj::Parser` is a big win in the performance arena. The
+isolation of options is another feature that should make life easier
+for developers.

data/pages/Rails.md

@@ -1,3 +1,15 @@
+# Rails Quickstart
+
+To universally replace Rails' use of the json gem with Oj, and also
+have Oj "take over" many methods on the JSON constant (`load`, `parse`, etc.) with
+their faster Oj counterparts, add this to an initializer:
+
+```ruby
+Oj.optimize_rails()
+```
+
+For more details and options, read on...
+
 # Oj Rails Compatibility
 
 The `:rails` mode mimics the ActiveSupport version 5 encoder. Rails and
@@ -41,7 +53,7 @@ The globals that ActiveSupport uses for encoding are:
 
 Those globals are aliased to also be accessed from the ActiveSupport module
 directly so `ActiveSupport::JSON::Encoding.time_precision` can also be accessed
-from `ActiveSupport.time_precision`. Oj makes use of these globals in mimicing
+from `ActiveSupport.time_precision`. Oj makes use of these globals in mimicking
 Rails after the `Oj::Rails.set_encode()` method is called. That also sets the
 `ActiveSupport.json_encoder` to the `Oj::Rails::Encoder` class.
 
@@ -125,7 +137,7 @@ gem 'oj', '3.7.12'
    Ruby which is used by the json gem and by Rails. Ruby varies the
    significant digits which can be either 16 or 17 depending on the value.
 
-2. Optimized Hashs do not collapse keys that become the same in the output. As
+2. Optimized Hashes do not collapse keys that become the same in the output. As
    an example, a non-String object that has a `to_s()` method will become the
    return value of the `to_s()` method in the output without checking to see if
    that has already been used. This could occur is a mix of String and Symbols

data/test/_test_active.rb

@@ -1,7 +1,7 @@
 #!/usr/bin/env ruby
-# encoding: UTF-8
+# frozen_string_literal: true
 
-$: << File.dirname(__FILE__)
+$LOAD_PATH << __dir__
 %w(lib ext test).each do |dir|
   $LOAD_PATH.unshift File.expand_path("../../#{dir}", __FILE__)
 end
@@ -13,14 +13,14 @@ require 'sqlite3'
 require 'active_record'
 require 'oj'
 
-#Oj.mimic_JSON()
+# Oj.mimic_JSON()
 Oj.default_options = {mode: :compat, indent: 2}
 
-#ActiveRecord::Base.logger = Logger.new(STDERR)
+# ActiveRecord::Base.logger = Logger.new(STDERR)
 
 ActiveRecord::Base.establish_connection(
-  :adapter => "sqlite3",
-  :database => ":memory:"
+  :adapter => 'sqlite3',
+  :database => ':memory:'
 )
 
 ActiveRecord::Schema.define do
@@ -37,8 +37,8 @@ end
 class ActiveTest < Minitest::Test
 
   def test_active
-    User.find_or_create_by(first_name: "John", last_name: "Smith", email: "john@example.com")
-    User.find_or_create_by(first_name: "Joan", last_name: "Smith", email: "joan@example.com")
+    User.find_or_create_by(first_name: 'John', last_name: 'Smith', email: 'john@example.com')
+    User.find_or_create_by(first_name: 'Joan', last_name: 'Smith', email: 'joan@example.com')
 
     # Single instance.
     assert_equal(%|{
@@ -71,6 +71,5 @@ class ActiveTest < Minitest::Test
 
     # Array of instances as json. (not Oj)
     assert_equal(%|[{"id":1,"first_name":"John","last_name":"Smith","email":"john@example.com"},{"id":2,"first_name":"Joan","last_name":"Smith","email":"joan@example.com"}]|, User.all.to_json)
-
   end
 end

data/test/_test_active_mimic.rb

@@ -1,7 +1,7 @@
 #!/usr/bin/env ruby
-# encoding: UTF-8
+# frozen_string_literal: true
 
-$: << File.dirname(__FILE__)
+$LOAD_PATH << __dir__
 %w(lib ext test).each do |dir|
   $LOAD_PATH.unshift File.expand_path("../../#{dir}", __FILE__)
 end
@@ -16,11 +16,11 @@ require 'oj'
 Oj.mimic_JSON()
 Oj.default_options = {mode: :compat, indent: 2}
 
-#ActiveRecord::Base.logger = Logger.new(STDERR)
+# ActiveRecord::Base.logger = Logger.new(STDERR)
 
 ActiveRecord::Base.establish_connection(
-  :adapter => "sqlite3",
-  :database => ":memory:"
+  :adapter => 'sqlite3',
+  :database => ':memory:'
 )
 
 ActiveRecord::Schema.define do
@@ -37,8 +37,8 @@ end
 class ActiveTest < Minitest::Test
 
   def test_active
-    User.find_or_create_by(first_name: "John", last_name: "Smith", email: "john@example.com")
-    User.find_or_create_by(first_name: "Joan", last_name: "Smith", email: "joan@example.com")
+    User.find_or_create_by(first_name: 'John', last_name: 'Smith', email: 'john@example.com')
+    User.find_or_create_by(first_name: 'Joan', last_name: 'Smith', email: 'joan@example.com')
 
     # Single instance.
     assert_equal(%|{
@@ -91,6 +91,5 @@ class ActiveTest < Minitest::Test
   }
 ]
 |, User.all.to_json)
-
   end
 end

data/test/_test_mimic_rails.rb

@@ -1,10 +1,10 @@
 #!/usr/bin/env ruby
-# encoding: UTF-8
+# frozen_string_literal: true
 
-$: << File.dirname(__FILE__)
+$LOAD_PATH << __dir__
 
 require 'helper'
-#Oj.mimic_JSON
+# Oj.mimic_JSON
 require 'rails/all'
 
 require 'active_model'
@@ -36,14 +36,12 @@ end
 class MimicRails < Minitest::Test
 
   def test_mimic_exception
-    begin
-      ActiveSupport::JSON.decode("{")
-      puts "Failed"
-    rescue ActiveSupport::JSON.parse_error
-      assert(true)
-    rescue Exception
-      assert(false, 'Expected a JSON::ParserError')
-    end
+    ActiveSupport::JSON.decode('{')
+    puts 'Failed'
+  rescue ActiveSupport::JSON.parse_error
+    assert(true)
+  rescue Exception
+    assert(false, 'Expected a JSON::ParserError')
   end
 
   def test_dump_string
@@ -84,18 +82,17 @@ class MimicRails < Minitest::Test
     category = Category.new(1, 'test')
     serializer = CategorySerializer.new(category)
 
-    json = serializer.to_json()
+    serializer.to_json()
     puts "*** serializer.to_json() #{serializer.to_json()}"
-    json = serializer.as_json()
+    serializer.as_json()
     puts "*** serializer.as_json() #{serializer.as_json()}"
-    json = JSON.dump(serializer)
+    JSON.dump(serializer)
     puts "*** JSON.dump(serializer) #{JSON.dump(serializer)}"
 
     puts "*** category.to_json() #{category.to_json()}"
     puts "*** category.as_json() #{category.as_json()}"
     puts "*** JSON.dump(serializer) #{JSON.dump(category)}"
     puts "*** Oj.dump(serializer) #{Oj.dump(category)}"
-
   end
 
   def test_dump_object_array
@@ -104,7 +101,7 @@ class MimicRails < Minitest::Test
     cat2 = Category.new(2, 'test')
     a = Array.wrap([cat1, cat2])
 
-    #serializer = CategorySerializer.new(a)
+    # serializer = CategorySerializer.new(a)
 
     puts "*** a.to_json() #{a.to_json()}"
     puts "*** a.as_json() #{a.as_json()}"
@@ -114,13 +111,13 @@ class MimicRails < Minitest::Test
 
   def test_dump_time
     Oj.default_options= {:indent => 2}
-    now = ActiveSupport::TimeZone['America/Chicago'].parse("2014-11-01 13:20:47")
+    now = ActiveSupport::TimeZone['America/Chicago'].parse('2014-11-01 13:20:47')
     json = Oj.dump(now, mode: :object, time_format: :xmlschema)
-    #puts "*** json: #{json}"
+    # puts "*** json: #{json}"
 
     oj_dump = Oj.load(json, mode: :object, time_format: :xmlschema)
-    #puts "Now: #{now}\n Oj: #{oj_dump}"
-    assert_equal("2014-11-01T13:20:47-05:00", oj_dump.xmlschema)
+    # puts "Now: #{now}\n Oj: #{oj_dump}"
+    assert_equal('2014-11-01T13:20:47-05:00', oj_dump.xmlschema)
   end
 
 end # MimicRails

data/test/activerecord/result_test.rb

@@ -13,13 +13,12 @@ Oj.default_options = { mode: :rails }
 
 class ActiveRecordResultTest < Minitest::Test
   def test_hash_rows
-
     result = ActiveRecord::Result.new(["one", "two"],
-				      [
-					["row 1 col 1", "row 1 col 2"],
-					["row 2 col 1", "row 2 col 2"],
-					["row 3 col 1", "row 3 col 2"],
-				      ])
+                                      [
+                                        ["row 1 col 1", "row 1 col 2"],
+                                        ["row 2 col 1", "row 2 col 2"],
+                                        ["row 3 col 1", "row 3 col 2"],
+                                      ])
     #puts "*** result: #{Oj.dump(result, indent: 2)}"
     json_result = if ActiveRecord.version >= Gem::Version.new("6")
                     result.to_a