oj 3.13.0 → 3.13.4

data/lib/oj/version.rb

@@ -1,5 +1,5 @@
 
 module Oj
   # Current version of the module.
-  VERSION = '3.13.0'
+  VERSION = '3.13.4'
 end

data/pages/Options.md

@@ -75,12 +75,22 @@ parse option to match the JSON gem. In that case either `Float`,
 If true Hash keys are cached or interned. There are trade-offs with
 caching keys. Large caches will use more memory and in extreme cases
 (like over a million) the cache may be slower than not using
-it. Repeated parsing of similar JSON docs is where cache_keys shines.
+it. Repeated parsing of similar JSON docs is where cache_keys shines
+especially with symbol keys.
+
+There is a maximum length for cached keys. Any key longer than 34
+bytes is not cached. Everything still works but the key is not cached.
 
 ### :cache_strings [Int]
 
 Shorter strings can be cached for better performance. A limit,
-cache_strings, defines the upper limit on what strings are cached.
+cache_strings, defines the upper limit on what strings are cached. As
+with cached keys only strings less than 35 bytes are cached even if
+the limit is set higher. Setting the limit to zero effectively
+disables the caching of string values.
+
+Note that caching for strings is for string values and not Hash keys
+or Object attributes.
 
 ### :circular [Boolean]
 
@@ -268,7 +278,7 @@ Use symbols instead of strings for hash keys.
 ### :symbolize_names [Boolean]
 
 Like :symbol_keys has keys are made into symbols but only when
-mimicing the JSON gem and then only as the JSON gem honors it so
+mimicking the JSON gem and then only as the JSON gem honors it so
 JSON.parse honors the option but JSON.load does not.
 
 ### :trace

data/pages/Parser.md

@@ -42,12 +42,12 @@ bleed over to other instances.
 
 ## How
 
-It's wonderfull to wish for a faster parser that solves all the
+It's wonderful to wish for a faster parser that solves all the
 annoyances of the previous parser but how was it done is a much more
 interesting question to answer.
 
 At the core, the API for parsing was changed. Instead of a sinle
-global parser any number of parsers can be created and each is seprate
+global parser any number of parsers can be created and each is separate
 from the others. The parser itself is able to rip through a JSON
 string, stream, or file and then make calls to a delegate to process
 the JSON elements according to the delegate behavior. This is similar
@@ -206,7 +206,7 @@ in array creation.
 
 For Hash the story is a little different. The bulk insert for Hash
 alternates keys and values but there is a wrinkle to consider. Since
-Ruby Object creation is triggered by the occurance of an element that
+Ruby Object creation is triggered by the occurrence of an element that
 matches a creation identifier the creation of a collection is not just
 for Array and Hash but also Object. Setting Object attributes uses an
 ID and not a VALUE. For that reason the keys should not be created as

data/pages/Rails.md

@@ -41,7 +41,7 @@ The globals that ActiveSupport uses for encoding are:
 
 Those globals are aliased to also be accessed from the ActiveSupport module
 directly so `ActiveSupport::JSON::Encoding.time_precision` can also be accessed
-from `ActiveSupport.time_precision`. Oj makes use of these globals in mimicing
+from `ActiveSupport.time_precision`. Oj makes use of these globals in mimicking
 Rails after the `Oj::Rails.set_encode()` method is called. That also sets the
 `ActiveSupport.json_encoder` to the `Oj::Rails::Encoder` class.
 
@@ -125,7 +125,7 @@ gem 'oj', '3.7.12'
    Ruby which is used by the json gem and by Rails. Ruby varies the
    significant digits which can be either 16 or 17 depending on the value.
 
-2. Optimized Hashs do not collapse keys that become the same in the output. As
+2. Optimized Hashes do not collapse keys that become the same in the output. As
    an example, a non-String object that has a `to_s()` method will become the
    return value of the `to_s()` method in the output without checking to see if
    that has already been used. This could occur is a mix of String and Symbols

data/test/benny.rb

@@ -0,0 +1,50 @@
+#!/usr/bin/env ruby
+# frozen_string_literal
+
+require 'bundler/inline'
+
+gemfile do
+  source 'https://rubygems.org'
+
+  gem 'oj'
+  gem 'benchmark-ips', require: 'benchmark/ips'
+end
+
+require 'json'
+require 'open-uri'
+
+CANADA_DATA_JSON = URI.parse('https://raw.githubusercontent.com/serde-rs/json-benchmark/master/data/canada.json').read
+CANADA_DATA = JSON.parse(CANADA_DATA_JSON)
+
+Benchmark.ips do |x|
+  x.config(:time => 10, :warmup => 5)
+
+  x.report("marshall Canada data with Oj") do
+    Oj.dump(CANADA_DATA)
+  end
+
+  x.report("marshall Canada data with JSON") do
+    JSON.dump(CANADA_DATA)
+  end
+
+  x.compare!
+end
+
+Oj.default_options = {
+  mode: :strict,
+  bigdecimal_load: :fast
+}
+
+Benchmark.ips do |x|
+  x.config(:time => 10, :warmup => 5)
+
+  x.report("unmarshall Canada data with Oj") do
+    Oj.load(CANADA_DATA_JSON)
+  end
+
+  x.report("unmarshall Canada data with JSON") do
+    JSON.parse(CANADA_DATA_JSON)
+  end
+
+  x.compare!
+end

data/test/big.rb

@@ -0,0 +1,15 @@
+#require 'active_support'
+#require 'active_support/core_ext'
+#require 'active_support/json'
+require 'oj'
+
+#Oj.optimize_rails
+Oj.mimic_JSON
+
+h = {:type=>:record, :name=>:group, :namespace=>"com.salsify.identity", :fields=>[{:name=>"id", :type=>{:name=>:salsify_uuid, :type=>:fixed, :namespace=>"com.salsify", :size=>38}}, {:name=>"type", :type=>"string", :default=>"groups"}, {:name=>"external_id", :type=>[:null, "string"], :default=>nil}, {:name=>"created_at", :type=>{:type=>"long", :logicalType=>"timestamp-micros"}}, {:name=>"updated_at", :type=>{:type=>"long", :logicalType=>"timestamp-micros"}}, {:name=>"name", :type=>"string"}, {:name=>"policy", :type=>[:null, {:type=>:record, :name=>:policy, :namespace=>"com.salsify.security", :fields=>[{:name=>"created_at", :type=>{:type=>"long", :logicalType=>"timestamp-micros"}}, {:name=>"updated_at", :type=>{:type=>"long", :logicalType=>"timestamp-micros"}}, {:name=>"id", :type=>"com.salsify.salsify_uuid"}, {:name=>"type", :type=>"string", :default=>"policies"}, {:name=>"external_id", :type=>[:null, "string"], :default=>nil}, {:name=>"name", :type=>"string"}, {:name=>"statements", :type=>{:type=>:array, :items=>{:type=>:record, :name=>:statement, :namespace=>"com.salsify.security", :fields=>[{:name=>"created_at", :type=>{:type=>"long", :logicalType=>"timestamp-micros"}}, {:name=>"updated_at", :type=>{:type=>"long", :logicalType=>"timestamp-micros"}}, {:name=>"id", :type=>"com.salsify.salsify_uuid"}, {:name=>"type", :type=>"string", :default=>"statements"}, {:name=>"external_id", :type=>[:null, "string"], :default=>nil}, {:name=>"action", :type=>{:name=>"__statement_action_enum", :type=>:enum, :namespace=>"com.salsify.security", :symbols=>[:manage, :read]}}, {:name=>"resource", :type=>{:name=>"__statement_resource_enum", :type=>:enum, :namespace=>"com.salsify.security", :symbols=>[:product, :digital_asset]}}, {:name=>"conditions", :type=>{:type=>:array, :items=>{:type=>:record, :name=>:condition, :namespace=>"com.salsify.security", :fields=>[{:name=>"created_at", :type=>{:type=>"long", :logicalType=>"timestamp-micros"}}, {:name=>"updated_at", :type=>{:type=>"long", :logicalType=>"timestamp-micros"}}, {:name=>"id", :type=>"com.salsify.salsify_uuid"}, {:name=>"type", :type=>"string", :default=>"conditions"}, {:name=>"external_id", :type=>[:null, "string"], :default=>nil}, {:name=>"operator", :type=>{:name=>"__condition_operator_enum", :type=>:enum, :namespace=>"com.salsify.security", :symbols=>[:equals]}}, {:name=>"attribute_type", :type=>{:name=>"__condition_attribute_type_enum", :type=>:enum, :namespace=>"com.salsify.security", :symbols=>[:resource]}}, {:name=>"value", :type=>"string"}, {:name=>"attribute", :type=>[:null, {:type=>:record, :name=>:reference, :namespace=>"com.salsify", :fields=>[{:name=>"id", :type=>"com.salsify.salsify_uuid"}, {:name=>"type", :type=>"string", :doc=>"snake_case, plural name for the resource type"}, {:name=>"external_id", :type=>[:null, "string"], :default=>nil}]}], :default=>nil}, {:name=>"broken", :type=>[:null, "boolean"], :default=>nil}]}}}]}}}]}], :default=>nil}]}
+
+#Oj.dump(h)
+puts JSON.pretty_generate(h)
+#puts JSON.fast_generate(h)
+#puts JSON.generate(h)
+

data/test/foo.rb

@@ -1,10 +1,8 @@
 #!/usr/bin/env ruby
 
-$: << File.dirname(__FILE__)
-$oj_dir = File.dirname(File.expand_path(File.dirname(__FILE__)))
-%w(lib ext).each do |dir|
-  $: << File.join($oj_dir, dir)
-end
+$: << '.'
+$: << '../lib'
+$: << '../ext'
 
 require 'oj'
 

data/test/json_gem/json_common_interface_test.rb

@@ -23,7 +23,7 @@ class JSONCommonInterfaceTest < Test::Unit::TestCase
       'h' => 1000.0,
       'i' => 0.001
     }
-    # Tired of chasing floating point rounding and precision. Oj not uses the
+    # Tired of chasing floating point rounding and precision. Oj now uses the
     # Ruby float parser in compat mode yet on i386 machines there are issues
     # with this test when the float is included.
     #@json = '{"a":2,"b":5.23683071,"c":"c","d":[1,"b",3.14],"e":{"foo":"bar"},'\

data/test/json_gem/json_generator_test.rb

@@ -113,7 +113,7 @@ EOT
   end
 
   # TBD Implement JSON.state to return state class.
-  # set state attibutes from defaults
+  # set state attributes from defaults
   # implement methods
   # circular should use circular in defaults or maybe always set to true, allow changes with [:check_circular]=
   def test_states

data/test/mem.rb

@@ -0,0 +1,33 @@
+#!/usr/bin/env ruby
+# encoding: UTF-8
+
+$: << '.'
+$: << File.join(File.dirname(__FILE__), "../lib")
+$: << File.join(File.dirname(__FILE__), "../ext")
+
+require 'oj'
+
+Oj.default_options = { mode: :rails, cache_keys: false, cache_str: -1 }
+
+def mem
+  `ps -o rss= -p #{$$}`.to_i
+end
+
+('a'..'z').each { |a|
+  ('a'..'z').each { |b|
+    ('a'..'z').each { |c|
+      ('a'..'z').each { |d|
+	('a'..'z').each { |e|
+	  ('a'..'z').each { |f|
+	    key = "#{a}#{b}#{c}#{d}#{e}#{f}"
+	    x = Oj.load(%|{ "#{key}": 101}|)
+	    #Oj.dump(x)
+	  }
+	}
+      }
+    }
+    puts "#{a}#{b} #{mem}"
+  }
+}
+
+Oj::Parser.new(:usual)

data/test/perf_once.rb

@@ -0,0 +1,58 @@
+#!/usr/bin/env ruby
+# encoding: UTF-8
+
+$: << '.'
+$: << File.join(File.dirname(__FILE__), "../lib")
+$: << File.join(File.dirname(__FILE__), "../ext")
+
+require 'oj'
+
+filename = 'tmp.json'
+File.open(filename, "w") { |f|
+  cnt = 0
+  f.puts('{')
+  ('a'..'z').each { |a|
+    ('a'..'z').each { |b|
+      ('a'..'z').each { |c|
+	('a'..'z').each { |d|
+	  f.puts(%|"#{a}#{b}#{c}#{d}":#{cnt},|)
+	  cnt += 1
+	}
+      }
+    }
+  }
+  f.puts('"_last":0}')
+}
+
+def mem
+  `ps -o rss= -p #{$$}`.to_i
+end
+
+Oj.default_options = { mode: :strict, cache_keys: false, cache_str: -1 }
+start = Time.now
+Oj.load_file('tmp.json')
+dur = Time.now - start
+GC.start
+puts "no cache duration: #{dur} @ #{mem}"
+
+Oj.default_options = { cache_keys: true }
+start = Time.now
+Oj.load_file('tmp.json')
+dur = Time.now - start
+GC.start
+puts "initial cache duration: #{dur} @ #{mem}"
+
+start = Time.now
+Oj.load_file('tmp.json')
+dur = Time.now - start
+GC.start
+puts "second cache duration: #{dur} @ #{mem}"
+
+10.times{ GC.start }
+start = Time.now
+Oj.load_file('tmp.json')
+dur = Time.now - start
+GC.start
+puts "after several GCs cache duration: #{dur} @ #{mem}"
+
+# TBD check memory use

data/test/perf_parser.rb

@@ -53,7 +53,7 @@ Oj.default_options = {create_id: '^', create_additions: true, class_cache: true}
 if $cache_keys
   Oj.default_options = {cache_keys: true, cache_str: 6, symbol_keys: $symbol_keys}
 else
-  Oj.default_options = {cache_keys: false, cache_str: 0, symbol_keys: $symbol_keys}
+  Oj.default_options = {cache_keys: false, cache_str: -1, symbol_keys: $symbol_keys}
 end
 JSON.parser = JSON::Ext::Parser
 
@@ -164,6 +164,11 @@ $obj_json = %|{
   "juliet": "junk"
 }|
 
+
+p_usual = Oj::Parser.new(:usual)
+p_usual.cache_keys = $cache_keys
+p_usual.cache_strings = ($cache_keys ? 6 : 0)
+p_usual.symbol_keys = $symbol_keys
 p_usual.create_id = '^'
 p_usual.class_cache = true
 p_usual.ignore_json_create = true

data/test/test_hash.rb

@@ -5,7 +5,7 @@ $: << File.dirname(__FILE__)
 
 require 'helper'
 
-class Hashi < Minitest::Test
+class HashTest < Minitest::Test
 
   module TestModule
   end

data/test/test_parser_memory.rb

@@ -0,0 +1,45 @@
+#!/usr/bin/env ruby
+# encoding: utf-8
+
+%w(lib ext test).each do |dir|
+  $LOAD_PATH.unshift File.expand_path("../../#{dir}", __FILE__)
+end
+
+require 'oj'
+
+json =<<-EOF
+{
+  "$id": "https://example.com/person.schema.json",
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "title": "Person",
+  "type": "object",
+  "properties": {
+    "firstName": {
+      "type": "string",
+      "description": "The person's first name."
+    },
+    "lastName": {
+      "type": "string",
+      "description": "The person's last name."
+    },
+    "age": {
+      "description": "Age in years which must be equal to or greater than zero.",
+      "type": "integer",
+      "minimum": 0
+    }
+  }
+}
+EOF
+
+#json = '{"abc": true}'
+#json = '[true,false]'
+
+100_001.times do |i|
+  p = Oj::Parser.new(:usual).parse(json)
+
+  if i % 10_000 == 0
+    GC.start
+    rss = Integer(`ps -o rss= -p #{Process.pid}`) / 1024.0
+    puts "#{i},#{rss} MB"
+  end
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: oj
 version: !ruby/object:Gem::Version
-  version: 3.13.0
+  version: 3.13.4
 platform: ruby
 authors:
 - Peter Ohler
-autorequire:
+autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-08-08 00:00:00.000000000 Z
+date: 2021-09-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake-compiler
@@ -91,6 +91,7 @@ extra_rdoc_files:
 - pages/Security.md
 - pages/WAB.md
 files:
+- CHANGELOG.md
 - LICENSE
 - README.md
 - ext/oj/buf.h
@@ -116,7 +117,6 @@ files:
 - ext/oj/err.h
 - ext/oj/extconf.rb
 - ext/oj/fast.c
-- ext/oj/hash_test.c
 - ext/oj/intern.c
 - ext/oj/intern.h
 - ext/oj/mimic_json.c
@@ -195,8 +195,8 @@ files:
 - test/activesupport6/test_common.rb
 - test/activesupport6/test_helper.rb
 - test/activesupport6/time_zone_test_helpers.rb
-- test/bar.rb
-- test/baz.rb
+- test/benny.rb
+- test/big.rb
 - test/files.rb
 - test/foo.rb
 - test/helper.rb
@@ -219,18 +219,19 @@ files:
 - test/json_gem/json_parser_test.rb
 - test/json_gem/json_string_matching_test.rb
 - test/json_gem/test_helper.rb
+- test/mem.rb
 - test/perf.rb
 - test/perf_compat.rb
 - test/perf_fast.rb
 - test/perf_file.rb
 - test/perf_object.rb
+- test/perf_once.rb
 - test/perf_parser.rb
 - test/perf_saj.rb
 - test/perf_scp.rb
 - test/perf_simple.rb
 - test/perf_strict.rb
 - test/perf_wab.rb
-- test/prec.rb
 - test/sample.rb
 - test/sample/change.rb
 - test/sample/dir.rb
@@ -257,6 +258,7 @@ files:
 - test/test_null.rb
 - test/test_object.rb
 - test/test_parser.rb
+- test/test_parser_memory.rb
 - test/test_parser_saj.rb
 - test/test_parser_usual.rb
 - test/test_rails.rb
@@ -269,7 +271,6 @@ files:
 - test/tests.rb
 - test/tests_mimic.rb
 - test/tests_mimic_addition.rb
-- test/zoo.rb
 homepage: http://www.ohler.com/oj
 licenses:
 - MIT
@@ -280,7 +281,7 @@ metadata:
   homepage_uri: http://www.ohler.com/oj/
   source_code_uri: https://github.com/ohler55/oj
   wiki_uri: https://github.com/ohler55/oj/wiki
-post_install_message:
+post_install_message: 
 rdoc_options:
 - "--title"
 - Oj
@@ -300,7 +301,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubygems_version: 3.2.22
-signing_key:
+signing_key: 
 specification_version: 4
 summary: A fast JSON parser and serializer.
 test_files:
@@ -324,8 +325,8 @@ test_files:
 - test/activesupport6/test_common.rb
 - test/activesupport6/test_helper.rb
 - test/activesupport6/time_zone_test_helpers.rb
-- test/bar.rb
-- test/baz.rb
+- test/benny.rb
+- test/big.rb
 - test/files.rb
 - test/foo.rb
 - test/helper.rb
@@ -348,18 +349,19 @@ test_files:
 - test/json_gem/json_parser_test.rb
 - test/json_gem/json_string_matching_test.rb
 - test/json_gem/test_helper.rb
+- test/mem.rb
 - test/perf.rb
 - test/perf_compat.rb
 - test/perf_fast.rb
 - test/perf_file.rb
 - test/perf_object.rb
+- test/perf_once.rb
 - test/perf_parser.rb
 - test/perf_saj.rb
 - test/perf_scp.rb
 - test/perf_simple.rb
 - test/perf_strict.rb
 - test/perf_wab.rb
-- test/prec.rb
 - test/sample/change.rb
 - test/sample/dir.rb
 - test/sample/doc.rb
@@ -386,6 +388,7 @@ test_files:
 - test/test_null.rb
 - test/test_object.rb
 - test/test_parser.rb
+- test/test_parser_memory.rb
 - test/test_parser_saj.rb
 - test/test_parser_usual.rb
 - test/test_rails.rb
@@ -398,4 +401,3 @@ test_files:
 - test/tests.rb
 - test/tests_mimic.rb
 - test/tests_mimic_addition.rb
-- test/zoo.rb