smarter_json 1.2.3 → 1.2.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: edd2cbc389a44f0714898f2a2942032632ca5f13a83d93acd708988866974893
-  data.tar.gz: 5cfa8719265797fd0fcd59f0060a5dd255aeba67ff229c78ab5696ad86f8b8e3
+  metadata.gz: 004e9e158ab0ec9ac6f7f3e532e01fac5ca525ed0f7409794052b6debbdeaa4e
+  data.tar.gz: deaa95b2c671d93db12c3059e79e4ab5908c44f009d5bc843b42b7561c030f7d
 SHA512:
-  metadata.gz: b0306cb7eab1db78053e8620f119b7debc54d836c38369b328bc0e0b9e8798d23eb4bbdb82281d036f62cb8cd296a3c8d7667f41a49d7e643a955830e08caf99
-  data.tar.gz: 0a5f052e115141014bcf2f0e031d43fb40521dd01da0f1ed5c6d4edeea785ead09f8ebcdfb76b580b4b8157391087ff2a79c5cd548ae7d8a0be96d0527a9f8fb
+  metadata.gz: 6b09e63686e0e37ecd25f8a1d511e865bb8f44df2fa95b213e3f5a34d4083ad5b30e0b15ce33c764cdaf960d7f2ebe6a859c8acb3ecdaa49c87118015ef34d88
+  data.tar.gz: 3ab34cf983f13f9b5479c69f4764c2cf44ba8da5290c1879b79c9de9338103025a67ab20bd6d6b25a1a5f8029b97868b53e8b8baae4daa430be370c046a222ad

data/CHANGELOG.md

@@ -13,6 +13,26 @@
 > ⚠️ We discourage the use of `process(input).first` / `process(input)[0]` because it silently drops potential additional documents
 >    Please use `process_one` if you are expecting only one JSON doc, e.g. in API payloads, because it emits on_warning if it finds multiple docs.
 
+## 1.2.5 (2026-07-01)
+
+RSpec tests: 1,282 → 1,308
+
+### New Features
+
+  - **RFC 7464 JSON Text Sequences are now read natively.** The record separator (`0x1E`) that frames each record is a first-class document separator, so an RS-framed stream parses into its documents without a warning, and bare-scalar records (a number / keyword / string on its own) are read correctly.
+
+## 1.2.4 (2026-07-01)
+
+RSpec tests: 1,268 → 1,282
+
+### Bug Fixes
+
+  - **Portable builds by default — fixes the "Illegal instruction" crash on mixed-hardware fleets.** The C extension was compiled with `-march=native` on every platform except Apple Silicon, baking in the build host's CPU instructions (e.g. AVX-512). A binary built on one machine could then crash with `Illegal instruction` when run on a CPU lacking those instructions — common when the build host differs from the run host (CI/build servers, Docker images, mixed-hardware fleets). The C extension is now built **portable** by default (no host-specific instructions).
+
+### New Features
+
+  - **`SMARTER_JSON_PERFORMANCE` build option** (`portable` default, `tuned`, or `max`) to opt into host-tuned or host-specific instructions at install time. See the [README](README.md#cpu-optimization-smarter_json_performance) / [Introduction](docs/_introduction.md#build-time-performance-tuning-smarter_json_performance) for details.
+
 ## 1.2.3 (2026-06-28)
 
 RSpec tests: 1,167 → 1,268

data/README.md

@@ -114,6 +114,26 @@ gem install smarter_json
 
 The C extension is built on install and used automatically. On platforms where it can't build, the pure-Ruby implementation runs instead and produces identical results.
 
+### CPU Optimization (`SMARTER_JSON_PERFORMANCE`)
+
+The C extension is compiled when the gem is installed. By default it is built **portable**: it uses no CPU-specific instructions, so a binary built on one machine runs on any other CPU of the same architecture. Set `SMARTER_JSON_PERFORMANCE` at install time to trade portability for speed:
+
+| Level                | Flags added                               | Portable?                        | Use when                              |
+|----------------------|-------------------------------------------|----------------------------------|---------------------------------------|
+| `portable` (default) | none                                      | Yes, any CPU of the arch         | Build host may differ from run host   |
+| `tuned`              | `-mtune=native`                           | Yes, instruction scheduling only | Build and run hosts share a microarch |
+| `max`                | `-march=native`, or `-mcpu=native` on ARM | No, host instruction optimization| Build host and run host are the same  |
+
+`max` enables host-specific instructions, so a binary built with it can crash with `Illegal instruction` if it later runs on a CPU that lacks them (for example, built on an AVX-512 machine and run on one without). `tuned` only changes instruction scheduling, never the instruction set, so it stays portable. Every flag is probed against your compiler at build time and skipped if unsupported, so an unavailable flag never breaks the build.
+
+```bash
+SMARTER_JSON_PERFORMANCE=tuned gem install smarter_json   # portable, tuned for this machine's microarchitecture
+SMARTER_JSON_PERFORMANCE=max   gem install smarter_json   # fastest, NOT portable — only when you build on the machine you run on
+SMARTER_JSON_PERFORMANCE=tuned bundle install             # same, under Bundler
+```
+
+For a fixed baseline instead of `native` (e.g. a portable-but-newer instruction set), pass flags directly via `CFLAGS`, which the build also honors: `CFLAGS="-march=x86-64-v2" gem install smarter_json`.
+
 ## Usage
 
 Pass a String of JSON content or an IO; you get back the extracted data. The same call handles strict JSON, JSON5, and HJSON-style config — there are no modes or flags.

data/docs/_introduction.md

@@ -37,6 +37,28 @@ It raises only on genuinely unreadable input (unterminated string, mismatched br
 
 Both the C extension and the pure-Ruby engine are **iterative, not recursive** — they track nesting on an explicit, heap-allocated stack rather than the call stack. So deeply nested input **cannot overflow the call stack or segfault**: nesting is bounded only by available memory, the same posture as Oj (the stdlib `json` caps at 100). The trade-off: there is currently **no fixed nesting or input-size limit**, so size-limit untrusted input upstream.
 
+## Build-Time Performance Tuning (`SMARTER_JSON_PERFORMANCE`)
+
+The C extension is compiled when the gem is installed. By default it is built **portable**: it uses no CPU-specific instructions, so a binary compiled on one machine runs on any other CPU of the same architecture. This matters whenever the machine that builds the gem differs from the machine that runs it — a CI or build server, a Docker image moved between hosts, or a mixed-hardware fleet. A build that bakes in instructions the run host lacks (such as AVX-512) would otherwise crash with `Illegal instruction`.
+
+Set `SMARTER_JSON_PERFORMANCE` at install time to trade portability for speed:
+
+| Level                | Flags added                               | Portable?                        | Use when                              |
+|----------------------|-------------------------------------------|----------------------------------|---------------------------------------|
+| `portable` (default) | none                                      | Yes, any CPU of the arch         | Build host may differ from run host   |
+| `tuned`              | `-mtune=native`                           | Yes, instruction scheduling only | Build and run hosts share a microarch |
+| `max`                | `-march=native`, or `-mcpu=native` on ARM | No, host instruction optimization| Build host and run host are the same  |
+
+`tuned` only changes instruction scheduling, never the instruction set, so it stays portable — and it pays off when the build and run hosts share a microarchitecture (the same chip, or a fleet of identical instances). `max` enables host-specific instructions and is the fastest, but a binary built with it can crash on a different CPU. Every flag is probed against your compiler at build time and skipped if unsupported, so an unavailable flag never breaks the build.
+
+```bash
+SMARTER_JSON_PERFORMANCE=tuned gem install smarter_json   # portable, tuned for this machine's microarchitecture
+SMARTER_JSON_PERFORMANCE=max   gem install smarter_json   # fastest, NOT portable — only when you build on the machine you run on
+SMARTER_JSON_PERFORMANCE=tuned bundle install             # same, under Bundler
+```
+
+For a fixed baseline instead of `native` (e.g. a portable-but-newer instruction set), pass flags directly via `CFLAGS`, which the build also honors: `CFLAGS="-march=x86-64-v2" gem install smarter_json`.
+
 ---------------
 
 NEXT: [The Basic Read API](./basic_read_api.md) | UP: [README](../README.md)

data/ext/smarter_json/cpu_flags.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+module SmarterJSON
+  # Pure (mkmf-free) selection of CPU-optimization flags from the
+  # SMARTER_JSON_PERFORMANCE environment variable. Kept separate from extconf.rb
+  # so the logic can be unit-tested without invoking a compiler.
+  #
+  # Levels:
+  #   portable - no host-specific flags. The binary runs on any CPU of the same
+  #              architecture. The safe default: a binary built here will not
+  #              crash with "Illegal instruction" on an older/different CPU.
+  #   tuned    - -mtune=native: tunes instruction scheduling for the build host's
+  #              microarchitecture WITHOUT changing the instruction set, so the
+  #              binary stays portable. A real win when build and run hosts share
+  #              a microarchitecture (same chip or a homogeneous fleet).
+  #   max      - host-specific instructions: -march=native, or -mcpu=native on
+  #              ARM/Clang where -march=native is rejected. Fastest, but NOT
+  #              portable -- may crash on a CPU lacking the build host's
+  #              instructions. Use only when build host and run host match.
+  #
+  # `accepts` is a predicate (in the real build, a wrapper over mkmf's
+  # try_compile) returning true when the compiler accepts a given flag; each
+  # candidate is probed so an unsupported flag is skipped rather than breaking
+  # the build.
+  module CpuFlags
+    LEVELS = %w[portable tuned max].freeze
+
+    # Candidate flags per level, in preference order. The first one the compiler
+    # accepts wins. `max` degrades march -> mcpu -> mtune; tuned only ever
+    # considers -mtune=native (never an instruction-set flag).
+    CANDIDATES = {
+      'portable' => [].freeze,
+      'tuned' => ['-mtune=native'].freeze,
+      'max' => ['-march=native', '-mcpu=native', '-mtune=native'].freeze,
+    }.freeze
+
+    # Returns a Hash: { level: String, flags: Array<String>, warning: String|nil }.
+    def self.select(raw_level, accepts:)
+      level, warning = normalize(raw_level)
+      chosen = CANDIDATES[level].find { |flag| accepts.call(flag) }
+      { level: level, flags: chosen ? [chosen] : [], warning: warning }
+    end
+
+    # Normalizes the env value to a known level. Unknown values fall back to
+    # 'portable' (a typo can then only ever be slower, never non-portable) and
+    # return a warning naming the bad value and the fallback.
+    def self.normalize(raw_level)
+      value = raw_level.to_s.strip.downcase
+      return ['portable', nil] if value.empty?
+      return [value, nil] if LEVELS.include?(value)
+
+      ['portable', "SMARTER_JSON_PERFORMANCE=#{raw_level.inspect} is not one of #{LEVELS.join('|')}; using 'portable'."]
+    end
+  end
+end

data/ext/smarter_json/extconf.rb

@@ -2,6 +2,7 @@
 
 require "mkmf"
 require "rbconfig"
+require_relative "cpu_flags"
 
 # Ruby sometimes ships CFLAGS with "-g -O3"; drop the debug half so the
 # extension is built optimized, not with debug info.
@@ -9,11 +10,31 @@ if RbConfig::MAKEFILE_CONFIG["CFLAGS"].include?("-g -O3")
   RbConfig::MAKEFILE_CONFIG["CFLAGS"] = RbConfig::MAKEFILE_CONFIG["CFLAGS"].sub("-g -O3", "-O3 $(cflags)")
 end
 
+# Probe whether the compiler accepts a flag by compiling a trivial program with
+# it. Lets us skip flags the toolchain rejects (e.g. -march=native on Clang/ARM,
+# or GCC-only flags on MSVC) instead of breaking the build. Replaces the old
+# RUBY_PLATFORM string guesses: ask the actual compiler, don't infer from the OS.
+def compiler_accepts?(flag)
+  try_compile("int main(void){return 0;}", flag)
+end
+
 optflags = "-O3 -flto -fomit-frame-pointer -DNDEBUG".dup
-# -march=native is skipped on arm64-darwin (Apple clang already targets the host).
-optflags << " -march=native" unless RUBY_PLATFORM.start_with?("arm64-darwin")
-# -fno-semantic-interposition: GCC/Clang only (not MSVC).
-optflags << " -fno-semantic-interposition" unless RUBY_PLATFORM.include?("mswin")
+
+# CPU optimization level, set via SMARTER_JSON_PERFORMANCE (default: portable).
+# See cpu_flags.rb for the full description of each level.
+#
+#   portable (default) - no host-specific flags; runs on any CPU of the same arch.
+#   tuned              - -mtune=native; host scheduling tuning, still portable.
+#   max                - host instruction set (-march/-mcpu native); fastest, but
+#                        NOT portable -- may crash on a CPU lacking those instructions.
+cpu = SmarterJSON::CpuFlags.select(ENV["SMARTER_JSON_PERFORMANCE"], accepts: method(:compiler_accepts?))
+warn(cpu[:warning]) if cpu[:warning]
+cpu[:flags].each { |flag| optflags << " #{flag}" }
+puts("SmarterJSON performance level: #{cpu[:level]} -- optflags: #{optflags}")
+
+# -fno-semantic-interposition: GCC/Clang only (not MSVC). Allows intra-library
+# calls to bypass the PLT on Linux and enables more aggressive LTO inlining.
+optflags << " -fno-semantic-interposition" if compiler_accepts?("-fno-semantic-interposition")
 
 CONFIG["optflags"]   = optflags
 CONFIG["debugflags"] = ""

data/ext/smarter_json/smarter_json.c

@@ -1624,14 +1624,17 @@ static int fj_implicit_root_ahead(fj_state *st) {
   return result;
 }
 
-/* Between top-level documents, whitespace, comments, AND commas all separate
- * (commas collapse like the in-container lenient-comma rule). A space alone never
- * separates — that is handled inside the document by the quoteless run. Mirrors
- * the Ruby Parser#skip_document_separators. */
+/* Between top-level documents, whitespace, comments, commas, AND the RFC 7464
+ * record separator (0x1E) all separate (commas collapse like the in-container
+ * lenient-comma rule; 0x1E frames each JSON Text Sequence record). A space alone
+ * never separates — that is handled inside the document by the quoteless run.
+ * Mirrors the Ruby Parser#skip_document_separators. */
 static void fj_skip_document_separators(fj_state *st) {
   for (;;) {
+    int b;
     fj_skip_ws_comments(st);
-    if (fj_byte(st) != ',') break;
+    b = fj_byte(st);
+    if (b != ',' && b != 0x1E) break;
     fj_advance(st, 1);
   }
 }
@@ -1640,8 +1643,9 @@ static int fj_is_hws(int b) { return b == ' ' || b == '\t' || b == 0x0B || b ==
 
 /* After a top-level value: a self-delimiting value (object / array / string) may be
  * followed by anything, but a bare scalar (number / keyword) must be followed by a
- * real separator — a newline, ',', a comment, or EOF. A space is NOT a separator, so
- * `1 2 3` and `42 "x" true` raise. Mirrors the Ruby Parser#enforce_scalar_boundary. */
+ * real separator — a newline, ',', the RFC 7464 record separator (0x1E), a comment,
+ * or EOF. A space is NOT a separator, so `1 2 3` and `42 "x" true` raise. Mirrors the
+ * Ruby Parser#enforce_scalar_boundary. */
 static void fj_enforce_scalar_boundary(fj_state *st, VALUE value) {
   int b, nx;
   if (RB_TYPE_P(value, T_STRING) || RB_TYPE_P(value, T_HASH) || RB_TYPE_P(value, T_ARRAY)) return;
@@ -1655,7 +1659,7 @@ static void fj_enforce_scalar_boundary(fj_state *st, VALUE value) {
     break;
   }
   b = fj_byte(st);
-  if (b == -1 || b == 0x0A || b == 0x0D || b == ',') return;
+  if (b == -1 || b == 0x0A || b == 0x0D || b == ',' || b == 0x1E) return;
   if (b == '#') return;
   if (b == '/') { nx = fj_byte_at(st, 1); if (nx == '/' || nx == '*') return; }
   fj_error(st, "a top-level number or keyword must be followed by a newline, ',', or end of input");

data/lib/smarter_json/parser.rb

@@ -384,6 +384,7 @@ module SmarterJSON
     TAB        = 0x09
     LF         = 0x0A
     CR         = 0x0D
+    RS         = 0x1E # RFC 7464 record separator — frames each JSON Text Sequence record
   end
 
   module Framer
@@ -1052,13 +1053,14 @@ module SmarterJSON
       parse_iter(implicit_root_object_ahead?)
     end
 
-    # Between top-level documents, whitespace, comments, AND commas all separate
-    # (commas collapse like the in-container lenient-comma rule). A space alone never
-    # separates — that is handled inside the document by the quoteless run, so
+    # Between top-level documents, whitespace, comments, commas, AND the RFC 7464
+    # record separator (0x1E) all separate (commas collapse like the in-container
+    # lenient-comma rule; 0x1E frames each JSON Text Sequence record). A space alone
+    # never separates — that is handled inside the document by the quoteless run, so
     # `1 2 3` is one document (the string "1 2 3") while `1, 2, 3` is three.
     def skip_document_separators
       skip_whitespace_and_comments
-      while byte == COMMA
+      while (b = byte) == COMMA || b == RS
         advance(1)
         skip_whitespace_and_comments
       end
@@ -1066,15 +1068,16 @@ module SmarterJSON
 
     # After a top-level value: a self-delimiting value (object / array / quoted string)
     # may be followed by anything (the next document self-delimits), but a bare scalar
-    # (number / keyword) must be followed by a real separator — a newline, ',', a
-    # comment, or EOF. A space is NOT a separator, so `1 2 3` and `42 "x" true` raise
-    # rather than silently splitting; bare top-level words raise in parse_value itself.
+    # (number / keyword) must be followed by a real separator — a newline, ',', the RFC
+    # 7464 record separator (0x1E), a comment, or EOF. A space is NOT a separator, so
+    # `1 2 3` and `42 "x" true` raise rather than silently splitting; bare top-level
+    # words raise in parse_value itself.
     def enforce_scalar_boundary(value)
       return if value.is_a?(String) || value.is_a?(Hash) || value.is_a?(Array)
 
       skip_horizontal_whitespace
       b = byte
-      return if b.nil? || b == LF || b == CR || b == COMMA
+      return if b.nil? || b == LF || b == CR || b == COMMA || b == RS
       return if b == HASH || (b == SLASH && ((c = byte_at(1)) == SLASH || c == STAR))
 
       raise error("a top-level number or keyword must be followed by a newline, ',', or end of input")

data/lib/smarter_json/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SmarterJSON
-  VERSION = "1.2.3"
+  VERSION = "1.2.5"
 end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: smarter_json
 version: !ruby/object:Gem::Version
-  version: 1.2.3
+  version: 1.2.5
 platform: ruby
 authors:
 - Tilo Sloboda
 bindir: exe
 cert_chain: []
-date: 2026-06-28 00:00:00.000000000 Z
+date: 2026-07-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bigdecimal
@@ -53,6 +53,7 @@ files:
 - docs/basic_write_api.md
 - docs/examples.md
 - docs/options.md
+- ext/smarter_json/cpu_flags.rb
 - ext/smarter_json/extconf.rb
 - ext/smarter_json/smarter_json.c
 - ext/smarter_json/smarter_json.h
@@ -91,7 +92,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.6.9
+rubygems_version: 4.0.15
 specification_version: 4
 summary: A lenient, fast JSON processor for Ruby — reads strict JSON, NDJSON, JSONL,
   JSON5, HJSON, and the messy JSON humans and LLMs actually write.