oj 3.9.1 → 3.16.11

data/README.md

@@ -1,11 +1,13 @@
 # [![{}j](http://www.ohler.com/dev/images/oj_comet_64.svg)](http://www.ohler.com/oj) gem
 
-[![Build Status](https://img.shields.io/travis/ohler55/oj/master.svg)](http://travis-ci.org/ohler55/oj?branch=master) [![AppVeyor](https://img.shields.io/appveyor/ci/ohler55/oj/master.svg)](https://ci.appveyor.com/project/ohler55/oj) ![Gem](https://img.shields.io/gem/v/oj.svg) ![Gem](https://img.shields.io/gem/dt/oj.svg) [![SemVer compatibility](https://api.dependabot.com/badges/compatibility_score?dependency-name=oj&package-manager=bundler&version-scheme=semver)](https://dependabot.com/compatibility-score.html?dependency-name=oj&package-manager=bundler&version-scheme=semver) [![TideLift](https://tidelift.com/badges/github/ohler55/oj)](https://tidelift.com/subscription/pkg/rubygems-oj?utm_source=rubygems-oj&utm_medium=referral&utm_campaign=readme)
+[![CI](https://github.com/ohler55/oj/actions/workflows/CI.yml/badge.svg)](https://github.com/ohler55/oj/actions/workflows/CI.yml)
+![Gem](https://img.shields.io/gem/v/oj.svg)
+![Gem](https://img.shields.io/gem/dt/oj.svg)
+[![TideLift](https://tidelift.com/badges/github/ohler55/oj)](https://tidelift.com/subscription/pkg/rubygems-oj?utm_source=rubygems-oj&utm_medium=referral&utm_campaign=readme)
 
 A *fast* JSON parser and Object marshaller as a Ruby gem.
 
-Version 3.0 is out! 3.0 provides better json gem and Rails compatibility. It
-also provides additional optimization options.
+Version 3.13 is out with a much faster parser (`Oj::Parser`) and option isolation.
 
 ## Using
 
@@ -40,9 +42,18 @@ or in Bundler:
 gem 'oj'
 ```
 
+## Rails and json quickstart
+
+See the Quickstart sections of the [Rails](pages/Rails.md) and [json](pages/JsonGem.md) docs.
+
+## multi_json
+
+Code which uses [multi_json](https://github.com/intridea/multi_json)
+will automatically prefer Oj if it is installed.
+
 ## Support
 
-[Get supported Oj with a Tidelift Subscription.](https://tidelift.com/subscription/pkg/rubygems-oj?utm_source=rubygems-oj&utm_medium=referral&utm_campaign=readme)
+[Get supported Oj with a Tidelift Subscription.](https://tidelift.com/subscription/pkg/rubygems-oj?utm_source=rubygems-oj&utm_medium=referral&utm_campaign=readme) Security updates are [supported](https://tidelift.com/security).
 
 ## Further Reading
 
@@ -50,7 +61,7 @@ For more details on options, modes, advanced features, and more follow these
 links.
 
  - [{file:Options.md}](pages/Options.md) for parse and dump options.
- - [{file:Modes.md}](pages/Modes.md) for details on modes for strict JSON compliance, mimicing the JSON gem, and mimicing Rails and ActiveSupport behavior.
+ - [{file:Modes.md}](pages/Modes.md) for details on modes for strict JSON compliance, mimicking the JSON gem, and mimicking Rails and ActiveSupport behavior.
  - [{file:JsonGem.md}](pages/JsonGem.md) includes more details on json gem compatibility and use.
  - [{file:Rails.md}](pages/Rails.md) includes more details on Rails and ActiveSupport compatibility and use.
  - [{file:Custom.md}](pages/Custom.md) includes more details on Custom mode.
@@ -58,10 +69,11 @@ links.
  - [{file:Compatibility.md}](pages/Compatibility.md) lists current compatibility with Rubys and Rails.
  - [{file:Advanced.md}](pages/Advanced.md) for fast parser and marshalling features.
  - [{file:Security.md}](pages/Security.md) for security considerations.
+ - [{file:InstallOptions.md}](pages/InstallOptions.md) for install option.
 
 ## Releases
 
-See [{file:CHANGELOG.md}](CHANGELOG.md)
+See [{file:CHANGELOG.md}](CHANGELOG.md) and [{file:RELEASE_NOTES.md}](RELEASE_NOTES.md)
 
 ## Links
 
@@ -97,8 +109,11 @@ Follow [@peterohler on Twitter](http://twitter.com/peterohler) for announcements
 
  - *Agoo-C, a high performance C web server supporting GraphQL on GitHub*: https://github.com/ohler55/agoo-c
 
+ - *oj-introspect, an example of creating an Oj parser extension in C*: https://github.com/meinac/oj-introspect
+
 #### Contributing
 
 + Provide a Pull Request off the `develop` branch.
 + Report a bug
 + Suggest an idea
++ Code is now formatted with the clang-format tool with the configuration file in the root of the repo.

data/RELEASE_NOTES.md

@@ -0,0 +1,61 @@
+# RELEASE NOTES
+
+The release notes here are organized by release. For a list of changes
+see the See [{file:CHANGELOG.md}](CHANGELOG.md) file. In this file are
+the steps to take to aid in keeping things rolling after updating to
+the latest version.
+
+## 3.13.7
+
+The default for JSON when mimicked by Oj is now to set
+`:allow_invalid_unicode`. To change that behavior JSON.load, set that
+option to false.
+
+## 3.13.x
+
+This release included a new cache that performs better than the
+earlier cache and a new high performance parser.
+
+### Cache
+
+The new cache includes a least recently used expiration to reduce
+memory use. The cache is also self adjusting and will expand as needed
+for better performance. It also handles Hash keys and string values
+with two options, `:cache_keys`, a boolean and `:cache_str` an
+integer. The `:cache_str` if set to more than zero is the limit for
+the length of string values to cache. The maximum value is 35 which
+allows strings up to 34 bytes to be cached.
+
+One interesting aspect of the cache is not so much the string caching
+which performs similar to the Ruby intern functions but the caching of
+symbols and object attribute names. There is a significant gain for
+symbols and object attributes.
+
+If the cache is not desired then setting the default options to turn
+it off can be done with this line:
+
+``` ruby
+Oj.default_options = { cache_keys: false, cache_str: 0 }
+```
+
+### Oj::Parser
+
+The new parser uses a different core that follows the approach taken
+by [OjC](https://github.com/ohler55/ojc) and
+[OjG](https://github.com/ohler55/ojg). It also takes advantage of the
+bulk Array and Hash functions. Another issue the new parser addresses
+is option management. Instead of a single global default_options each
+parser instance maintains it's own options.
+
+There is a price to be paid when using the Oj::Parser. The API is not
+the same the older parser. A single parser can only be used in a
+single thread. This allows reuse of internal buffers for additional
+improvements in performance.
+
+The performane advantage of the Oj::Parse is that it is more than 3
+times faster than the Oj::compat_load call and 6 times faster than the
+JSON gem.
+
+### Dump Performance
+
+Thanks to Watson1978 Oj.dump also received a speed boost.

data/ext/oj/buf.h

@@ -1,99 +1,81 @@
-/* buf.h
- * Copyright (c) 2011, Peter Ohler
- * All rights reserved.
- * 
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
- * 
- *  - Redistributions of source code must retain the above copyright notice, this
- *    list of conditions and the following disclaimer.
- * 
- *  - Redistributions in binary form must reproduce the above copyright notice,
- *    this list of conditions and the following disclaimer in the documentation
- *    and/or other materials provided with the distribution.
- * 
- *  - Neither the name of Peter Ohler nor the names of its contributors may be
- *    used to endorse or promote products derived from this software without
- *    specific prior written permission.
- * 
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
- * DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
- * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
- * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
- * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
+// Copyright (c) 2011 Peter Ohler. All rights reserved.
+// Licensed under the MIT License. See LICENSE file in the project root for license details.
 
 #ifndef OJ_BUF_H
 #define OJ_BUF_H
 
+#include "mem.h"
 #include "ruby.h"
 
 typedef struct _buf {
-    char	*head;
-    char	*end;
-    char	*tail;
-    char	base[1024];
+    char *head;
+    char *end;
+    char *tail;
+    char  base[1024];
 } *Buf;
 
-inline static void
-buf_init(Buf buf) {
+inline static void buf_init(Buf buf) {
     buf->head = buf->base;
-    buf->end = buf->base + sizeof(buf->base) - 1;
+    buf->end  = buf->base + sizeof(buf->base) - 1;
     buf->tail = buf->head;
 }
 
-inline static void
-buf_cleanup(Buf buf) {
+inline static void buf_reset(Buf buf) {
+    buf->tail = buf->head;
+}
+
+inline static void buf_cleanup(Buf buf) {
     if (buf->base != buf->head) {
-        xfree(buf->head);
+        OJ_R_FREE(buf->head);
     }
 }
 
-inline static size_t
-buf_len(Buf buf) {
+inline static size_t buf_len(Buf buf) {
     return buf->tail - buf->head;
 }
 
-inline static void
-buf_append_string(Buf buf, const char *s, size_t slen) {
+inline static const char *buf_str(Buf buf) {
+    *buf->tail = '\0';
+    return buf->head;
+}
+
+inline static void buf_append_string(Buf buf, const char *s, size_t slen) {
+    if (0 == slen) {
+        return;
+    }
+
     if (buf->end <= buf->tail + slen) {
-	size_t	len = buf->end - buf->head;
-	size_t	toff = buf->tail - buf->head;
-	size_t	new_len = len + slen + len / 2;
+        size_t len     = buf->end - buf->head;
+        size_t toff    = buf->tail - buf->head;
+        size_t new_len = len + slen + len / 2;
 
-	if (buf->base == buf->head) {
-	    buf->head = ALLOC_N(char, new_len);
-	    memcpy(buf->head, buf->base, len);
-	} else {
-	    REALLOC_N(buf->head, char, new_len);
-	}
-	buf->tail = buf->head + toff;
-	buf->end = buf->head + new_len - 1;
+        if (buf->base == buf->head) {
+            buf->head = OJ_R_ALLOC_N(char, new_len);
+            memcpy(buf->head, buf->base, len);
+        } else {
+            OJ_R_REALLOC_N(buf->head, char, new_len);
+        }
+        buf->tail = buf->head + toff;
+        buf->end  = buf->head + new_len - 1;
     }
     memcpy(buf->tail, s, slen);
     buf->tail += slen;
 }
-    
-inline static void
-buf_append(Buf buf, char c) {
+
+inline static void buf_append(Buf buf, char c) {
     if (buf->end <= buf->tail) {
-	size_t	len = buf->end - buf->head;
-	size_t	toff = buf->tail - buf->head;
-	size_t	new_len = len + len / 2;
+        size_t len     = buf->end - buf->head;
+        size_t toff    = buf->tail - buf->head;
+        size_t new_len = len + len / 2;
 
-	if (buf->base == buf->head) {
-	    buf->head = ALLOC_N(char, new_len);
-	    memcpy(buf->head, buf->base, len);
-	} else {
-	    REALLOC_N(buf->head, char, new_len);
-	}
-	buf->tail = buf->head + toff;
-	buf->end = buf->head + new_len - 1;
+        if (buf->base == buf->head) {
+            buf->head = OJ_R_ALLOC_N(char, new_len);
+            memcpy(buf->head, buf->base, len);
+        } else {
+            OJ_R_REALLOC_N(buf->head, char, new_len);
+        }
+        buf->tail = buf->head + toff;
+        buf->end  = buf->head + new_len - 1;
     }
     *buf->tail = c;
     buf->tail++;

data/ext/oj/cache.c

@@ -0,0 +1,329 @@
+// Copyright (c) 2011, 2021 Peter Ohler. All rights reserved.
+// Licensed under the MIT License. See LICENSE file in the project root for license details.
+
+#if HAVE_PTHREAD_MUTEX_INIT
+#include <pthread.h>
+#endif
+#include <stdlib.h>
+
+#include "cache.h"
+#include "mem.h"
+
+// The stdlib calloc, realloc, and free are used instead of the Ruby ALLOC,
+// ALLOC_N, REALLOC, and xfree since the later could trigger a GC which will
+// either corrupt memory or if the mark function locks will deadlock.
+
+#define REHASH_LIMIT 4
+#define MIN_SHIFT 8
+#define REUSE_MAX 8192
+
+#if HAVE_PTHREAD_MUTEX_INIT
+#define CACHE_LOCK(c) pthread_mutex_lock(&((c)->mutex))
+#define CACHE_UNLOCK(c) pthread_mutex_unlock(&((c)->mutex))
+#else
+#define CACHE_LOCK(c) rb_mutex_lock((c)->mutex)
+#define CACHE_UNLOCK(c) rb_mutex_unlock((c)->mutex)
+#endif
+
+// almost the Murmur hash algorithm
+#define M 0x5bd1e995
+
+typedef struct _slot {
+    struct _slot     *next;
+    VALUE             val;
+    uint64_t          hash;
+    volatile uint32_t use_cnt;
+    uint8_t           klen;
+    char              key[CACHE_MAX_KEY];
+} *Slot;
+
+typedef struct _cache {
+    volatile Slot  *slots;
+    volatile size_t cnt;
+    VALUE (*form)(const char *str, size_t len);
+    uint64_t size;
+    uint64_t mask;
+    VALUE (*intern)(struct _cache *c, const char *key, size_t len);
+    volatile Slot reuse;
+    size_t        rcnt;
+#if HAVE_PTHREAD_MUTEX_INIT
+    pthread_mutex_t mutex;
+#else
+    VALUE mutex;
+#endif
+    uint8_t xrate;
+    bool    mark;
+} *Cache;
+
+void cache_set_form(Cache c, VALUE (*form)(const char *str, size_t len)) {
+    c->form = form;
+}
+
+static uint64_t hash_calc(const uint8_t *key, size_t len) {
+    const uint8_t *end     = key + len;
+    const uint8_t *endless = key + (len & 0xFFFFFFFC);
+    uint64_t       h       = (uint64_t)len;
+    uint64_t       k;
+
+    while (key < endless) {
+        k = (uint64_t)*key++;
+        k |= (uint64_t)*key++ << 8;
+        k |= (uint64_t)*key++ << 16;
+        k |= (uint64_t)*key++ << 24;
+
+        k *= M;
+        k ^= k >> 24;
+        h *= M;
+        h ^= k * M;
+    }
+    if (1 < end - key) {
+        uint16_t k16 = (uint16_t)*key++;
+
+        k16 |= (uint16_t)*key++ << 8;
+        h ^= k16 << 8;
+    }
+    if (key < end) {
+        h ^= *key;
+    }
+    h *= M;
+    h ^= h >> 13;
+    h *= M;
+    h ^= h >> 15;
+
+    return h;
+}
+
+static void rehash(Cache c) {
+    uint64_t osize;
+    Slot    *end;
+    Slot    *sp;
+
+    osize    = c->size;
+    c->size  = osize * 4;
+    c->mask  = c->size - 1;
+    c->slots = OJ_REALLOC((void *)c->slots, sizeof(Slot) * c->size);
+    memset((Slot *)c->slots + osize, 0, sizeof(Slot) * osize * 3);
+    end = (Slot *)c->slots + osize;
+    for (sp = (Slot *)c->slots; sp < end; sp++) {
+        Slot s    = *sp;
+        Slot next = NULL;
+
+        *sp = NULL;
+        for (; NULL != s; s = next) {
+            uint64_t h      = s->hash & c->mask;
+            Slot    *bucket = (Slot *)c->slots + h;
+
+            next    = s->next;
+            s->next = *bucket;
+            *bucket = s;
+        }
+    }
+}
+
+static VALUE lockless_intern(Cache c, const char *key, size_t len) {
+    uint64_t       h      = hash_calc((const uint8_t *)key, len);
+    Slot          *bucket = (Slot *)c->slots + (h & c->mask);
+    Slot           b;
+    volatile VALUE rkey;
+
+    while (REUSE_MAX < c->rcnt) {
+        if (NULL != (b = c->reuse)) {
+            c->reuse = b->next;
+            OJ_FREE(b);
+            c->rcnt--;
+        } else {
+            // An accounting error occured somewhere so correct it.
+            c->rcnt = 0;
+        }
+    }
+    for (b = *bucket; NULL != b; b = b->next) {
+        if ((uint8_t)len == b->klen && 0 == strncmp(b->key, key, len)) {
+            b->use_cnt += 16;
+            return b->val;
+        }
+    }
+    rkey = c->form(key, len);
+    if (NULL == (b = c->reuse)) {
+        b = OJ_CALLOC(1, sizeof(struct _slot));
+    } else {
+        c->reuse = b->next;
+        c->rcnt--;
+    }
+    b->hash = h;
+    memcpy(b->key, key, len);
+    b->klen     = (uint8_t)len;
+    b->key[len] = '\0';
+    b->val      = rkey;
+    b->use_cnt  = 4;
+    b->next     = *bucket;
+    *bucket     = b;
+    c->cnt++;  // Don't worry about wrapping. Worse case is the entry is removed and recreated.
+    if (REHASH_LIMIT < c->cnt / c->size) {
+        rehash(c);
+    }
+    return rkey;
+}
+
+static VALUE locking_intern(Cache c, const char *key, size_t len) {
+    uint64_t       h;
+    Slot          *bucket;
+    Slot           b;
+    uint64_t       old_size;
+    volatile VALUE rkey;
+
+    CACHE_LOCK(c);
+    while (REUSE_MAX < c->rcnt) {
+        if (NULL != (b = c->reuse)) {
+            c->reuse = b->next;
+            OJ_FREE(b);
+            c->rcnt--;
+        } else {
+            // An accounting error occured somewhere so correct it.
+            c->rcnt = 0;
+        }
+    }
+    h      = hash_calc((const uint8_t *)key, len);
+    bucket = (Slot *)c->slots + (h & c->mask);
+    for (b = *bucket; NULL != b; b = b->next) {
+        if ((uint8_t)len == b->klen && 0 == strncmp(b->key, key, len)) {
+            b->use_cnt += 4;
+            CACHE_UNLOCK(c);
+
+            return b->val;
+        }
+    }
+    old_size = c->size;
+    // The creation of a new value may trigger a GC which be a problem if the
+    // cache is locked so make sure it is unlocked for the key value creation.
+    if (NULL != (b = c->reuse)) {
+        c->reuse = b->next;
+        c->rcnt--;
+    }
+    CACHE_UNLOCK(c);
+    if (NULL == b) {
+        b = OJ_CALLOC(1, sizeof(struct _slot));
+    }
+    rkey    = c->form(key, len);
+    b->hash = h;
+    memcpy(b->key, key, len);
+    b->klen     = (uint8_t)len;
+    b->key[len] = '\0';
+    b->val      = rkey;
+    b->use_cnt  = 16;
+
+    // Lock again to add the new entry.
+    CACHE_LOCK(c);
+    if (old_size != c->size) {
+        h      = hash_calc((const uint8_t *)key, len);
+        bucket = (Slot *)c->slots + (h & c->mask);
+    }
+    b->next = *bucket;
+    *bucket = b;
+    c->cnt++;  // Don't worry about wrapping. Worse case is the entry is removed and recreated.
+    if (REHASH_LIMIT < c->cnt / c->size) {
+        rehash(c);
+    }
+    CACHE_UNLOCK(c);
+
+    return rkey;
+}
+
+Cache cache_create(size_t size, VALUE (*form)(const char *str, size_t len), bool mark, bool locking) {
+    Cache c     = OJ_CALLOC(1, sizeof(struct _cache));
+    int   shift = 0;
+
+    for (; REHASH_LIMIT < size; size /= 2, shift++) {
+    }
+    if (shift < MIN_SHIFT) {
+        shift = MIN_SHIFT;
+    }
+#if HAVE_PTHREAD_MUTEX_INIT
+    pthread_mutex_init(&c->mutex, NULL);
+#else
+    c->mutex = rb_mutex_new();
+#endif
+    c->size  = 1 << shift;
+    c->mask  = c->size - 1;
+    c->slots = OJ_CALLOC(c->size, sizeof(Slot));
+    c->form  = form;
+    c->xrate = 1;  // low
+    c->mark  = mark;
+    if (locking) {
+        c->intern = locking_intern;
+    } else {
+        c->intern = lockless_intern;
+    }
+    return c;
+}
+
+void cache_set_expunge_rate(Cache c, int rate) {
+    c->xrate = (uint8_t)rate;
+}
+
+void cache_free(void *data) {
+    Cache    c = (Cache)data;
+    uint64_t i;
+
+    for (i = 0; i < c->size; i++) {
+        Slot next;
+        Slot s;
+
+        for (s = c->slots[i]; NULL != s; s = next) {
+            next = s->next;
+            OJ_FREE(s);
+        }
+    }
+    OJ_FREE((void *)c->slots);
+    OJ_FREE(c);
+}
+
+void cache_mark(void *data) {
+    Cache    c = (Cache)data;
+    uint64_t i;
+
+#if !HAVE_PTHREAD_MUTEX_INIT
+    rb_gc_mark(c->mutex);
+#endif
+    if (0 == c->cnt) {
+        return;
+    }
+    for (i = 0; i < c->size; i++) {
+        Slot s;
+        Slot prev = NULL;
+        Slot next;
+
+        for (s = c->slots[i]; NULL != s; s = next) {
+            next = s->next;
+            if (0 == s->use_cnt) {
+                if (NULL == prev) {
+                    c->slots[i] = next;
+                } else {
+                    prev->next = next;
+                }
+                c->cnt--;
+                s->next  = c->reuse;
+                c->reuse = s;
+                c->rcnt++;
+                continue;
+            }
+            switch (c->xrate) {
+            case 0: break;
+            case 2: s->use_cnt -= 2; break;
+            case 3: s->use_cnt /= 2; break;
+            default: s->use_cnt--; break;
+            }
+            if (c->mark) {
+                rb_gc_mark(s->val);
+            }
+            prev = s;
+        }
+    }
+}
+
+VALUE
+cache_intern(Cache c, const char *key, size_t len) {
+    if (CACHE_MAX_KEY <= len) {
+        return c->form(key, len);
+    }
+    return c->intern(c, key, len);
+}

data/ext/oj/cache.h

@@ -0,0 +1,22 @@
+// Copyright (c) 2021 Peter Ohler. All rights reserved.
+// Licensed under the MIT License. See LICENSE file in the project root for license details.
+
+#ifndef CACHE_H
+#define CACHE_H
+
+#include <ruby.h>
+#include <stdbool.h>
+
+#define CACHE_MAX_KEY 35
+
+struct _cache;
+typedef struct _cache *Cache;
+
+extern struct _cache *cache_create(size_t size, VALUE (*form)(const char *str, size_t len), bool mark, bool locking);
+extern void           cache_free(void *data);
+extern void           cache_mark(void *data);
+extern void           cache_set_form(struct _cache *c, VALUE (*form)(const char *str, size_t len));
+extern VALUE          cache_intern(struct _cache *c, const char *key, size_t len);
+extern void           cache_set_expunge_rate(struct _cache *c, int rate);
+
+#endif /* CACHE_H */