smarter_json 1.2.2 → 1.2.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a31a7485e53042c06cb28c92f23e24cb55067fd312704c0c9ac0776868f20293
-  data.tar.gz: 531187f8f7ea38f573785f09fa6e217f63a4393fd537d80c92f00973aeb2b375
+  metadata.gz: 1bdfb87113872a6659af0063f196de84f9074b81eedc90ae63445fa6a9f86a0a
+  data.tar.gz: 1271826d3097455921c7f058710ea42651daa75a0de9438b0f0d581e7abc8b7a
 SHA512:
-  metadata.gz: f88266d06416277355f770645d694be6e81a8faef22a86806cd1e4620e773dee50aeeeeb09cfbc0de47354972250c2654e599c5c03ba0926ca129d9d6a638b0c
-  data.tar.gz: c8bdd4a34d4617c897c815f2526c0f5555459b35bad47106a1db2b71dedac5d7a2605d0a5f92676735ae4895208fcdf1272f05dce09026aa696b84f143693ee5
+  metadata.gz: 4885f735c132bea79affc74ddb82504c581b722a03f52fbbbaabe27fa818347715bf1753b2fc539fb86991448a3e7987719d679f18210aaab54afbe06ee1d021
+  data.tar.gz: c9fafa246a337eb025a089e3db4909c596d52b023b5b06291e7f3b8114f225c57ad4e48b4fd742ffd295db1c95e6cd9574e68adc6bba0d9f5a4c8695b0620649

data/CHANGELOG.md

@@ -13,9 +13,32 @@
 > ⚠️ We discourage the use of `process(input).first` / `process(input)[0]` because it silently drops potential additional documents
 >    Please use `process_one` if you are expecting only one JSON doc, e.g. in API payloads, because it emits on_warning if it finds multiple docs.
 
-## 1.2.2 (unreleased)
+## 1.2.4 (2026-07-01)
 
-RSpec tests: 1,167
+RSpec tests: 1,268 → 1,282
+
+### Bug Fixes
+
+  - **Portable builds by default — fixes the "Illegal instruction" crash on mixed-hardware fleets.** The C extension was compiled with `-march=native` on every platform except Apple Silicon, baking in the build host's CPU instructions (e.g. AVX-512). A binary built on one machine could then crash with `Illegal instruction` when run on a CPU lacking those instructions — common when the build host differs from the run host (CI/build servers, Docker images, mixed-hardware fleets). The C extension is now built **portable** by default (no host-specific instructions).
+
+### New Features
+
+  - **`SMARTER_JSON_PERFORMANCE` build option** (`portable` default, `tuned`, or `max`) to opt into host-tuned or host-specific instructions at install time. See the [README](README.md#cpu-optimization-smarter_json_performance) / [Introduction](docs/_introduction.md#build-time-performance-tuning-smarter_json_performance) for details.
+
+## 1.2.3 (2026-06-28)
+
+RSpec tests: 1,167 → 1,268
+
+Fixing some encoding corner cases:
+  - **UTF-16 / UTF-32** and **Shift_JIS** (and other CJK double-byte encodings such as Big5 / GBK / GB18030) previously raised or mis-parsed; they now parse, with string values tagged in the input's encoding.
+  - Applies to String, file (`process_file`), and IO / streaming (`foreach`) input — including a file the caller opened with transcoding (e.g. `File.open(path, "r:UTF-8:UTF-16LE")`), where the output is the encoding the bytes arrive in.
+  - Streaming a **Latin-1 / Windows-1252** (or other single-byte) file or IO now preserves that encoding too, instead of mislabelling or raising.
+  - Streaming a UTF-16 / UTF-32 / Shift_JIS (or transcoding) source via `foreach` / `process_file` is now **bounded-memory** — it frames and parses one document at a time instead of reading the whole input into memory.
+  - New `:replace_char` option (default `"?"`): when a `\uXXXX` escape decodes to a character the input's encoding can't represent (e.g. an emoji inside a Shift_JIS document), that character is replaced rather than raising. `replace_char: ""` drops it.
+
+## 1.2.2 (2026-06-19)
+
+RSpec tests: 1,165 → 1,167
 
 - The Eisel-Lemire fast path for `decimal_precision: :float` now covers decimals with **up to 19 significant digits** (was 18). 19 digits is the most that fits exactly in a `uint64` (max 19-digit ≈ 1.0e19 < `UINT64_MAX` ≈ 1.8e19), so these no longer fall back to the slower `strtod`. Still correctly rounded, bit-for-bit identical to the stdlib — verified across 18/19-digit round-to-even tie shapes.
 

data/README.md

@@ -114,6 +114,26 @@ gem install smarter_json
 
 The C extension is built on install and used automatically. On platforms where it can't build, the pure-Ruby implementation runs instead and produces identical results.
 
+### CPU Optimization (`SMARTER_JSON_PERFORMANCE`)
+
+The C extension is compiled when the gem is installed. By default it is built **portable**: it uses no CPU-specific instructions, so a binary built on one machine runs on any other CPU of the same architecture. Set `SMARTER_JSON_PERFORMANCE` at install time to trade portability for speed:
+
+| Level                | Flags added                               | Portable?                        | Use when                              |
+|----------------------|-------------------------------------------|----------------------------------|---------------------------------------|
+| `portable` (default) | none                                      | Yes, any CPU of the arch         | Build host may differ from run host   |
+| `tuned`              | `-mtune=native`                           | Yes, instruction scheduling only | Build and run hosts share a microarch |
+| `max`                | `-march=native`, or `-mcpu=native` on ARM | No, host instruction optimization| Build host and run host are the same  |
+
+`max` enables host-specific instructions, so a binary built with it can crash with `Illegal instruction` if it later runs on a CPU that lacks them (for example, built on an AVX-512 machine and run on one without). `tuned` only changes instruction scheduling, never the instruction set, so it stays portable. Every flag is probed against your compiler at build time and skipped if unsupported, so an unavailable flag never breaks the build.
+
+```bash
+SMARTER_JSON_PERFORMANCE=tuned gem install smarter_json   # portable, tuned for this machine's microarchitecture
+SMARTER_JSON_PERFORMANCE=max   gem install smarter_json   # fastest, NOT portable — only when you build on the machine you run on
+SMARTER_JSON_PERFORMANCE=tuned bundle install             # same, under Bundler
+```
+
+For a fixed baseline instead of `native` (e.g. a portable-but-newer instruction set), pass flags directly via `CFLAGS`, which the build also honors: `CFLAGS="-march=x86-64-v2" gem install smarter_json`.
+
 ## Usage
 
 Pass a String of JSON content or an IO; you get back the extracted data. The same call handles strict JSON, JSON5, and HJSON-style config — there are no modes or flags.
@@ -222,6 +242,7 @@ In short: **SmarterJSON's C path matches or beats Oj/strict on every file** (app
 | `decimal_precision` | `:auto`      | `:auto` keeps high-precision decimals as `BigDecimal`; `:float` forces `Float`; `:bigdecimal` forces `BigDecimal` |
 | `acceleration`    | `true`       | `true` uses the C extension when compiled and loadable; `false` forces pure Ruby (identical results) |
 | `encoding`        | `nil`        | labels the input's encoding; `nil` keeps the input's own (no transcoding pass; see below) |
+| `replace_char`    | `"?"`        | replacement for a char a `\uXXXX` escape decodes to that the input's encoding can't represent (e.g. an emoji inside Shift_JIS); `""` drops it |
 | `on_warning`      | `nil`        | a callable invoked once per lenient fix applied (`:empty_slot`, `:empty_value`, `:duplicate_key`, `:number_overflow`), passed a `SmarterJSON::Warning`; the return value is never changed. See below. |
 
 ## Examples
@@ -358,6 +379,8 @@ TEXT
 
 `encoding:` (default `nil`) labels what the input is — it does **not** transcode. With `nil`, SmarterJSON keeps the input's own encoding tag and emits string values with that same tag, the way `smarter_csv` does — **with one smart default:** input tagged `ASCII-8BIT` (BINARY) whose bytes are valid UTF-8 is treated as UTF-8. That is exactly how `Net::HTTP` and many HTTP libraries hand you a `response.body` (correct UTF-8 bytes, BINARY tag); without this, string values would come back tagged `ASCII-8BIT` and compare unequal to UTF-8 literals. If such `ASCII-8BIT` input is *not* valid UTF-8, it raises `SmarterJSON::EncodingError` rather than guess a legacy encoding — pass an explicit `encoding:` (e.g. `"ISO-8859-1"`) for that. Bytes invalid for an explicitly claimed encoding also raise `SmarterJSON::EncodingError` (a kind of `SmarterJSON::ParseError`).
 
+UTF-16 / UTF-32, Shift_JIS, and other CJK double-byte encodings parse too: SmarterJSON works on a UTF-8 copy internally and re-tags the result back into the input's own encoding, so values come back in the encoding the bytes arrived in (a UTF-16 / UTF-32 BOM is consumed on the way in). The one edge case — a `\uXXXX` escape that decodes to a character that encoding can't represent (e.g. an emoji inside a Shift_JIS document) — is replaced by `replace_char` (default `"?"`, or `""` to drop it) rather than raising.
+
 ## Nesting & untrusted input
 
 Both the C extension and the pure-Ruby engine are **iterative, not recursive** — they track nesting on an explicit, heap-allocated stack rather than the call stack. So deeply nested input **cannot overflow the call stack or segfault**: nesting is bounded only by available memory, the same posture as Oj (which also ships no nesting limit; the stdlib `json` caps at 100). The `deeply_nested.json` benchmark (212 MB of nesting) is handled without issue. **`generate` is iterative too**, so serializing a deeply nested Ruby structure can't overflow the stack either — reading *and* writing are both depth-safe.

data/docs/_introduction.md

@@ -37,6 +37,28 @@ It raises only on genuinely unreadable input (unterminated string, mismatched br
 
 Both the C extension and the pure-Ruby engine are **iterative, not recursive** — they track nesting on an explicit, heap-allocated stack rather than the call stack. So deeply nested input **cannot overflow the call stack or segfault**: nesting is bounded only by available memory, the same posture as Oj (the stdlib `json` caps at 100). The trade-off: there is currently **no fixed nesting or input-size limit**, so size-limit untrusted input upstream.
 
+## Build-Time Performance Tuning (`SMARTER_JSON_PERFORMANCE`)
+
+The C extension is compiled when the gem is installed. By default it is built **portable**: it uses no CPU-specific instructions, so a binary compiled on one machine runs on any other CPU of the same architecture. This matters whenever the machine that builds the gem differs from the machine that runs it — a CI or build server, a Docker image moved between hosts, or a mixed-hardware fleet. A build that bakes in instructions the run host lacks (such as AVX-512) would otherwise crash with `Illegal instruction`.
+
+Set `SMARTER_JSON_PERFORMANCE` at install time to trade portability for speed:
+
+| Level                | Flags added                               | Portable?                        | Use when                              |
+|----------------------|-------------------------------------------|----------------------------------|---------------------------------------|
+| `portable` (default) | none                                      | Yes, any CPU of the arch         | Build host may differ from run host   |
+| `tuned`              | `-mtune=native`                           | Yes, instruction scheduling only | Build and run hosts share a microarch |
+| `max`                | `-march=native`, or `-mcpu=native` on ARM | No, host instruction optimization| Build host and run host are the same  |
+
+`tuned` only changes instruction scheduling, never the instruction set, so it stays portable — and it pays off when the build and run hosts share a microarchitecture (the same chip, or a fleet of identical instances). `max` enables host-specific instructions and is the fastest, but a binary built with it can crash on a different CPU. Every flag is probed against your compiler at build time and skipped if unsupported, so an unavailable flag never breaks the build.
+
+```bash
+SMARTER_JSON_PERFORMANCE=tuned gem install smarter_json   # portable, tuned for this machine's microarchitecture
+SMARTER_JSON_PERFORMANCE=max   gem install smarter_json   # fastest, NOT portable — only when you build on the machine you run on
+SMARTER_JSON_PERFORMANCE=tuned bundle install             # same, under Bundler
+```
+
+For a fixed baseline instead of `native` (e.g. a portable-but-newer instruction set), pass flags directly via `CFLAGS`, which the build also honors: `CFLAGS="-march=x86-64-v2" gem install smarter_json`.
+
 ---------------
 
 NEXT: [The Basic Read API](./basic_read_api.md) | UP: [README](../README.md)

data/docs/options.md

@@ -22,6 +22,7 @@ These options are passed to [`SmarterJSON.process`](./basic_read_api.md), `Smart
 | `:duplicate_key`  | `:last_wins` | How to handle a key that repeats within one object: `:last_wins` or `:first_wins`. (Every repeat is also reported through `:on_warning` — see below.)                          |
 | `:encoding`       | `nil`        | Labels the input's encoding (e.g. `"UTF-8"`). It does **not** trigger a transcoding pass — see below.                  |
 | `:on_warning`     | `nil`        | A callable invoked once per lenient fix applied, passed a `SmarterJSON::Warning`. Never changes the return value. See below. |
+| `:replace_char`   | `"?"`        | Replacement for a character a `\uXXXX` escape decodes to that the input's encoding can't represent (e.g. an emoji inside Shift_JIS). `""` drops it. See below. |
 | `:symbolize_keys` | `false`      | Return object keys as Symbols instead of Strings.                                                                      |
 
 ```ruby
@@ -47,7 +48,11 @@ The warning types are `:empty_slot` (a collapsed empty comma slot, e.g. `[1,,2]`
 
 ### A note on `:encoding`
 
-`:encoding` labels what the input *is* — it does not transcode. With the default `nil`, SmarterJSON keeps the input's own encoding tag and emits string values with that tag, the same way `smarter_csv` handles encodings — **with one smart default:** input tagged `ASCII-8BIT` (BINARY) that is valid UTF-8 is treated as UTF-8. This is how `Net::HTTP` returns a `response.body`; without it, those string values would compare unequal to UTF-8 literals. `ASCII-8BIT` input that is *not* valid UTF-8 raises `SmarterJSON::EncodingError` — pass an explicit `:encoding` (e.g. `"ISO-8859-1"`) for genuinely-legacy bytes. Bytes invalid for an explicitly claimed encoding also raise `SmarterJSON::EncodingError` (a kind of `SmarterJSON::ParseError`). A UTF-8 BOM is handled automatically; UTF-16 / UTF-32 input is out of scope.
+`:encoding` labels what the input *is* — it does not transcode. With the default `nil`, SmarterJSON keeps the input's own encoding tag and emits string values with that tag, the same way `smarter_csv` handles encodings — **with one smart default:** input tagged `ASCII-8BIT` (BINARY) that is valid UTF-8 is treated as UTF-8. This is how `Net::HTTP` returns a `response.body`; without it, those string values would compare unequal to UTF-8 literals. `ASCII-8BIT` input that is *not* valid UTF-8 raises `SmarterJSON::EncodingError` — pass an explicit `:encoding` (e.g. `"ISO-8859-1"`) for genuinely-legacy bytes. Bytes invalid for an explicitly claimed encoding also raise `SmarterJSON::EncodingError` (a kind of `SmarterJSON::ParseError`). A UTF-8 BOM is handled automatically. UTF-16 / UTF-32, Shift_JIS, and other CJK double-byte encodings are now supported as well: the document parses and string values come back tagged in the input's own encoding (a UTF-16 / UTF-32 BOM is consumed on the way in). The one wrinkle — a `\uXXXX` escape that decodes to a character the input's encoding can't represent — is handled by `:replace_char` (above).
+
+### A note on `:replace_char`
+
+For an input in an encoding that can't be byte-scanned directly (UTF-16 / UTF-32, Shift_JIS, and other CJK double-byte encodings), SmarterJSON parses a UTF-8 copy and re-tags the result back into the input's encoding, so you get values in the encoding the bytes arrived in. A `\uXXXX` escape can decode to a character that encoding can't represent — e.g. an emoji inside a Shift_JIS document. Rather than raise, that single character is replaced by `:replace_char` (default `"?"`). Set `replace_char: ""` to drop it, or pass any string your target encoding can hold (e.g. the geta mark `"〓"` for Shift_JIS). It applies only on this transcode-and-re-tag path; for plain UTF-8 / single-byte input it never comes into play.
 
 ### A note on `:decimal_precision`
 

data/ext/smarter_json/cpu_flags.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+module SmarterJSON
+  # Pure (mkmf-free) selection of CPU-optimization flags from the
+  # SMARTER_JSON_PERFORMANCE environment variable. Kept separate from extconf.rb
+  # so the logic can be unit-tested without invoking a compiler.
+  #
+  # Levels:
+  #   portable - no host-specific flags. The binary runs on any CPU of the same
+  #              architecture. The safe default: a binary built here will not
+  #              crash with "Illegal instruction" on an older/different CPU.
+  #   tuned    - -mtune=native: tunes instruction scheduling for the build host's
+  #              microarchitecture WITHOUT changing the instruction set, so the
+  #              binary stays portable. A real win when build and run hosts share
+  #              a microarchitecture (same chip or a homogeneous fleet).
+  #   max      - host-specific instructions: -march=native, or -mcpu=native on
+  #              ARM/Clang where -march=native is rejected. Fastest, but NOT
+  #              portable -- may crash on a CPU lacking the build host's
+  #              instructions. Use only when build host and run host match.
+  #
+  # `accepts` is a predicate (in the real build, a wrapper over mkmf's
+  # try_compile) returning true when the compiler accepts a given flag; each
+  # candidate is probed so an unsupported flag is skipped rather than breaking
+  # the build.
+  module CpuFlags
+    LEVELS = %w[portable tuned max].freeze
+
+    # Candidate flags per level, in preference order. The first one the compiler
+    # accepts wins. `max` degrades march -> mcpu -> mtune; tuned only ever
+    # considers -mtune=native (never an instruction-set flag).
+    CANDIDATES = {
+      'portable' => [].freeze,
+      'tuned' => ['-mtune=native'].freeze,
+      'max' => ['-march=native', '-mcpu=native', '-mtune=native'].freeze,
+    }.freeze
+
+    # Returns a Hash: { level: String, flags: Array<String>, warning: String|nil }.
+    def self.select(raw_level, accepts:)
+      level, warning = normalize(raw_level)
+      chosen = CANDIDATES[level].find { |flag| accepts.call(flag) }
+      { level: level, flags: chosen ? [chosen] : [], warning: warning }
+    end
+
+    # Normalizes the env value to a known level. Unknown values fall back to
+    # 'portable' (a typo can then only ever be slower, never non-portable) and
+    # return a warning naming the bad value and the fallback.
+    def self.normalize(raw_level)
+      value = raw_level.to_s.strip.downcase
+      return ['portable', nil] if value.empty?
+      return [value, nil] if LEVELS.include?(value)
+
+      ['portable', "SMARTER_JSON_PERFORMANCE=#{raw_level.inspect} is not one of #{LEVELS.join('|')}; using 'portable'."]
+    end
+  end
+end

data/ext/smarter_json/extconf.rb

@@ -2,6 +2,7 @@
 
 require "mkmf"
 require "rbconfig"
+require_relative "cpu_flags"
 
 # Ruby sometimes ships CFLAGS with "-g -O3"; drop the debug half so the
 # extension is built optimized, not with debug info.
@@ -9,11 +10,31 @@ if RbConfig::MAKEFILE_CONFIG["CFLAGS"].include?("-g -O3")
   RbConfig::MAKEFILE_CONFIG["CFLAGS"] = RbConfig::MAKEFILE_CONFIG["CFLAGS"].sub("-g -O3", "-O3 $(cflags)")
 end
 
+# Probe whether the compiler accepts a flag by compiling a trivial program with
+# it. Lets us skip flags the toolchain rejects (e.g. -march=native on Clang/ARM,
+# or GCC-only flags on MSVC) instead of breaking the build. Replaces the old
+# RUBY_PLATFORM string guesses: ask the actual compiler, don't infer from the OS.
+def compiler_accepts?(flag)
+  try_compile("int main(void){return 0;}", flag)
+end
+
 optflags = "-O3 -flto -fomit-frame-pointer -DNDEBUG".dup
-# -march=native is skipped on arm64-darwin (Apple clang already targets the host).
-optflags << " -march=native" unless RUBY_PLATFORM.start_with?("arm64-darwin")
-# -fno-semantic-interposition: GCC/Clang only (not MSVC).
-optflags << " -fno-semantic-interposition" unless RUBY_PLATFORM.include?("mswin")
+
+# CPU optimization level, set via SMARTER_JSON_PERFORMANCE (default: portable).
+# See cpu_flags.rb for the full description of each level.
+#
+#   portable (default) - no host-specific flags; runs on any CPU of the same arch.
+#   tuned              - -mtune=native; host scheduling tuning, still portable.
+#   max                - host instruction set (-march/-mcpu native); fastest, but
+#                        NOT portable -- may crash on a CPU lacking those instructions.
+cpu = SmarterJSON::CpuFlags.select(ENV["SMARTER_JSON_PERFORMANCE"], accepts: method(:compiler_accepts?))
+warn(cpu[:warning]) if cpu[:warning]
+cpu[:flags].each { |flag| optflags << " #{flag}" }
+puts("SmarterJSON performance level: #{cpu[:level]} -- optflags: #{optflags}")
+
+# -fno-semantic-interposition: GCC/Clang only (not MSVC). Allows intra-library
+# calls to bypass the PLT on Linux and enables more aggressive LTO inlining.
+optflags << " -fno-semantic-interposition" if compiler_accepts?("-fno-semantic-interposition")
 
 CONFIG["optflags"]   = optflags
 CONFIG["debugflags"] = ""

data/lib/smarter_json/options.rb

@@ -12,6 +12,7 @@ module SmarterJSON
       duplicate_key: :last_wins, # :last_wins | :first_wins  (repeats are also reported via on_warning)
       decimal_precision: :auto,  # :auto | :float | :bigdecimal  (Oj-compatible decimal handling)
       on_warning: nil,           # a callable invoked once per non-fatal lenient fix (a SmarterJSON::Warning)
+      replace_char: "?",         # replacement for a char not representable in the input's encoding (undef: :replace); "" drops it
     }.freeze
 
     module_function
@@ -56,6 +57,9 @@ module SmarterJSON
       unless encoding.nil? || encoding.is_a?(String)
         errors << "encoding must be nil or a String (got #{encoding.class})"
       end
+      unless options[:replace_char].is_a?(String)
+        errors << "replace_char must be a String (got #{options[:replace_char].class})"
+      end
 
       raise ArgumentError, "SmarterJSON: invalid options — #{errors.join('; ')}" if errors.any?
 

data/lib/smarter_json/parser.rb

@@ -41,22 +41,42 @@ module SmarterJSON
 
   # SmarterJSON.process_file(path, options = {}) — open a file and process it.
   #
-  # The :encoding option labels the file's encoding (default "UTF-8"); it does NOT
-  # trigger a transcoding pass — the parser works on the bytes in their native
-  # encoding and emits string values with the same encoding tag. With a block,
-  # streams document-by-document straight from disk in bounded memory (never
-  # loading the whole file); the documents are read as newline-delimited
-  # (NDJSON / JSONL), one per line.
+  # The :encoding option labels the file's encoding (default "UTF-8").
+  #
+  # The user can send any encoding to SmarterJSON - we make zero assumptions about encoding.
+  # We also do not "normalize" the input to a different encoding on our own (this is not Python).
+  #
+  # We parse the bytes in whatever encoding they arrive in and emit string values
+  # with that same encoding tag.
+  #
+  # The caller is free to transcode the input themselves (e.g. open the file with a "r:ext:int" mode);
+  # however the bytes arrive, we parse them and preserve their encoding. With a block,
+  # streams document-by-document straight from disk in bounded memory (neverloading the whole file);
+  # the documents are read as newline-delimited (NDJSON / JSONL), one per line.
+  #
   def process_file(path, options = {}, &block)
     options = Options.process_options(options)
     encoding = options[:encoding] || "UTF-8"
+    mode = file_read_mode(encoding)
     if block
-      File.open(path, "r:#{encoding}") { |io| stream_io(io, options, &block) }
+      File.open(path, mode) { |io| stream_io(io, options, &block) }
     else
-      process(File.read(path, encoding: encoding), options)
+      process(File.read(path, mode: mode), options)
     end
   end
 
+  # Read mode for process_file. Binary mode is required for ASCII-incompatible encodings
+  # (UTF-16 / UTF-32) — text mode refuses them ("ASCII incompatible encoding needs binmode").
+  # ASCII-compatible encodings keep TEXT mode, so newline translation (e.g. \r\n on Windows)
+  # is unchanged — binmode only applies where text mode is impossible anyway.
+  def file_read_mode(encoding)
+    incompatible = encoding.to_s.split(":").any? do |name|
+      enc = Encoding.find(name) rescue nil
+      enc && !enc.ascii_compatible?
+    end
+    incompatible ? "rb:#{encoding}" : "r:#{encoding}"
+  end
+
   # SmarterJSON.foreach(source, options = {}) — the streaming, composable sibling of
   # process_file, mirroring the stdlib convention (CSV.foreach / File.foreach): a
   # plain Enumerator (NOT Enumerator::Lazy), so .map / .select behave the normal way
@@ -163,11 +183,121 @@ module SmarterJSON
     raise EncodingError, "input is tagged ASCII-8BIT and is not valid UTF-8 — pass encoding: to declare its encoding"
   end
 
+  # Legacy CJK double-byte encodings whose trail bytes can fall in the ASCII range, so a
+  # 0x5C trail byte looks like a string escape, a 0x7B like a brace, etc. — i.e. they are
+  # ascii_compatible? yet still NOT safe to byte-scan for JSON structure. (EUC-* and
+  # single-byte encodings keep their non-ASCII bytes above 0x7F, so they ARE safe.)
+  UNSCANNABLE_ASCII_COMPATIBLE = %w[
+    Shift_JIS Windows-31J MacJapanese SHIFT_JISX0213 SJIS-DoCoMo SJIS-KDDI SJIS-SoftBank
+    Big5 Big5-HKSCS Big5-UAO CP950 GBK GB18030 GB12345
+  ].each_with_object({}) do |name, h|
+    h[Encoding.find(name)] = true
+  rescue ArgumentError
+    # encoding not built into this Ruby — skip it
+  end.freeze
+
+  # True when an Encoding cannot be scanned directly for JSON structure — the non
+  # ASCII-compatible ones (UTF-16 / UTF-32, where structure is in code units) and the CJK
+  # double-byte ones above. For these we parse a UTF-8 copy and emit the values back in the
+  # original encoding. (Over-including a safe encoding only costs a transcode round-trip; the
+  # result is still correct.)
+  def unscannable_enc?(enc)
+    return true unless enc.ascii_compatible?
+
+    UNSCANNABLE_ASCII_COMPATIBLE.key?(enc)
+  end
+
+  # The encoding the bytes arrived in when they must be parsed via a UTF-8 copy (see
+  # unscannable_enc?); nil when the bytes are directly byte-scannable.
+  def unscannable_encoding(input)
+    enc = input.encoding
+    unscannable_enc?(enc) ? enc : nil
+  end
+
+  # Generic UTF-16 / UTF-32 prepend a byte-order mark to EVERY string when you encode TO them.
+  # Map the generic encoding to the concrete endianness (read from the input's own BOM) so the
+  # re-tagged values are BOM-free and usable. Concrete and non-Unicode encodings pass through.
+  def concrete_unicode_encoding(input, enc)
+    return enc unless enc == Encoding::UTF_16 || enc == Encoding::UTF_32
+
+    head = input.byteslice(0, 4).to_s.b
+    if enc == Encoding::UTF_16
+      head.start_with?("\xFF\xFE".b) ? Encoding::UTF_16LE : Encoding::UTF_16BE
+    else
+      head.start_with?("\xFF\xFE\x00\x00".b) ? Encoding::UTF_32LE : Encoding::UTF_32BE
+    end
+  end
+
+  # Transcode the input to a UTF-8 working copy for scanning. Invalid bytes raise the gem's
+  # own SmarterJSON::EncodingError (not a bare Ruby Encoding error), matching the rest.
+  def to_utf8_copy(input)
+    input.encode(Encoding::UTF_8)
+  rescue Encoding::UndefinedConversionError, Encoding::InvalidByteSequenceError
+    raise EncodingError, "invalid byte sequence for #{input.encoding.name}"
+  end
+
+  # Re-tag one scalar into `enc`. A character not representable in `enc` (e.g. an emoji from a
+  # `\u` escape inside a Shift_JIS document) is replaced by `replace` (the :replace_char option,
+  # default "?") — uniform encoding, never raises. (`invalid:` can't trigger here: the value
+  # came from a valid UTF-8 parse.)
+  def reencode_scalar(obj, enc, replace)
+    return obj unless obj.is_a?(String)
+
+    obj.encode(enc, invalid: :replace, undef: :replace, replace: replace)
+  end
+
+  # Re-tag a parsed value's strings (Hash keys/values, Array elements, nested) into `enc`,
+  # so we emit values in the encoding the bytes arrived in after parsing a UTF-8 copy.
+  # ITERATIVE (an explicit work stack, not recursion) so a deeply nested document is
+  # depth-safe — like the parser itself — and can't raise SystemStackError.
+  def deep_encode(root, enc, replace)
+    return reencode_scalar(root, enc, replace) unless root.is_a?(Array) || root.is_a?(Hash)
+
+    out = root.is_a?(Array) ? [] : {}
+    stack = [[root, out]]
+    until stack.empty?
+      src, dst = stack.pop
+      if src.is_a?(Array)
+        src.each do |v|
+          child = (v.is_a?(Array) ? [] : {}) if v.is_a?(Array) || v.is_a?(Hash)
+          dst << (child || reencode_scalar(v, enc, replace))
+          stack.push([v, child]) if child
+        end
+      else
+        src.each do |k, v|
+          key = reencode_scalar(k, enc, replace)
+          child = (v.is_a?(Array) ? [] : {}) if v.is_a?(Array) || v.is_a?(Hash)
+          dst[key] = child || reencode_scalar(v, enc, replace)
+          stack.push([v, child]) if child
+        end
+      end
+    end
+    out
+  end
+
   # Stream documents from an IO incrementally, yielding each recovered top-level
   # document without slurping the whole input into memory first.
   def stream_io(io, options, &block)
+    ext = io.respond_to?(:external_encoding) ? io.external_encoding : nil
+    int = io.respond_to?(:internal_encoding) ? io.internal_encoding : nil
+    out_enc = int || ext # the encoding the caller expects in the output
+    source  = ext        # the encoding readpartial's raw bytes are actually in
+
+    # The Framer reads via readpartial, which returns ASCII-8BIT — it drops the IO's encoding
+    # and ignores transcoding. When the byte-scanner can't frame the raw bytes directly — they
+    # are in an unscannable encoding (UTF-16 / UTF-32 / Shift_JIS / ...), or the IO transcodes
+    # (the wanted output encoding differs from the on-wire bytes) — transcode each chunk to a
+    # UTF-8 view, frame documents there one at a time, and emit each in `out_enc`. Bounded
+    # memory: only one document is buffered, never the whole stream.
+    if source && out_enc && (unscannable_enc?(source) || out_enc != source)
+      return stream_transcoded(io, source, out_enc, options, &block)
+    end
+
     count = 0
     Framer.each_document(io) do |doc|
+      # readpartial dropped the IO's encoding tag; restore it so a Latin-1 / Windows-1252 /
+      # etc. stream is parsed and emitted in its own encoding, not mislabelled.
+      doc = doc.dup.force_encoding(out_enc) if out_enc && doc.encoding != out_enc
       # Recovery.process_string yields each value and returns how many it yielded;
       # blank / comment-only framed segments yield none, so count tracks actual
       # documents (== values yielded), not raw framed segments.
@@ -176,6 +306,25 @@ module SmarterJSON
     count
   end
 
+  # Bounded-memory streaming for an unscannable or transcoding IO (see stream_io). Each chunk
+  # is transcoded to a UTF-8 view and framed there one document at a time; each framed document
+  # is parsed and emitted in `out_enc` — the same parse-then-re-tag path as the whole-buffer
+  # case, but per document, so peak memory is bounded by one document, not the whole stream.
+  def stream_transcoded(io, source, out_enc, options, &block)
+    first   = Framer.read_chunk(io)
+    out_enc = concrete_unicode_encoding(first.to_s, out_enc) # generic UTF-16/32 -> concrete via BOM
+    # No converter when the raw bytes are already UTF-8 (e.g. a UTF-8 -> UTF-16 transcoding IO):
+    # the bytes need no transcoding to be byte-scanned, only the OUTPUT is re-tagged (deep_encode).
+    conv    = [Encoding::UTF_8, Encoding::US_ASCII].include?(source) ? nil : Encoding::Converter.new(source, Encoding::UTF_8)
+    opts    = options.merge(encoding: nil)
+    replace = options[:replace_char]
+    count   = 0
+    Framer.each_document_transcoded(io, conv, first) do |utf8_doc|
+      count += Recovery.process_string(utf8_doc, opts) { |v| block.call(deep_encode(v, out_enc, replace)) }
+    end
+    count
+  end
+
   # process_one's "more than one document" notice — routed to on_warning if the caller
   # gave one, else Rails.logger when Rails is loaded, else Kernel#warn. Never silent,
   # never raised.
@@ -192,7 +341,10 @@ module SmarterJSON
     end
   end
 
-  private_class_method :process_content, :stream_io, :warn_extra_documents
+  private_class_method :process_content, :stream_io, :stream_transcoded, :warn_extra_documents,
+                       :file_read_mode, :normalize_default_encoding, :unscannable_enc?,
+                       :unscannable_encoding, :concrete_unicode_encoding, :to_utf8_copy,
+                       :reencode_scalar, :deep_encode
 
   # Named byte values, shared by the Parser FSM and the Framer / Recovery byte
   # scanners so none of them spell out raw hex. Included where needed.
@@ -261,6 +413,59 @@ module SmarterJSON
       yield buffer unless separators_only?(buffer)
     end
 
+    # Like each_document, but the IO's raw bytes are in `conv`'s source encoding (UTF-16 /
+    # UTF-32 / Shift_JIS / ...): each chunk is transcoded to a UTF-8 view and framed there, so
+    # the byte-level splitter works. `first_chunk` is the already-read first raw chunk (the
+    # caller sniffs a BOM from it). Memory stays bounded by one document, like each_document.
+    def each_document_transcoded(io, conv, first_chunk)
+      buffer = +""
+      scan = 0
+      doc_start = nil
+      stack = []
+      mode = nil
+
+      raw = first_chunk
+      while raw
+        chunk = transcode_chunk(conv, raw)
+        unless chunk.empty?
+          buffer << chunk
+          loop do
+            emitted, buffer, scan, doc_start, stack, mode = scan_buffer(buffer, scan, doc_start, stack, mode)
+            break unless emitted
+
+            yield emitted
+          end
+        end
+        raw = read_chunk(io)
+      end
+
+      finish_transcode(conv) # truncated / invalid trailing bytes -> SmarterJSON::EncodingError
+
+      yield buffer unless separators_only?(buffer)
+    end
+
+    # Push one raw chunk through the converter, returning the UTF-8 produced so far. An
+    # incomplete trailing multibyte sequence is held inside the converter until the next chunk;
+    # invalid bytes raise SmarterJSON::EncodingError (matching the whole-buffer to_utf8_copy).
+    def transcode_chunk(conv, raw)
+      return raw.dup.force_encoding(Encoding::UTF_8) if conv.nil? # raw bytes are already UTF-8
+
+      out = +""
+      status = conv.primitive_convert(raw.dup, out, nil, nil, partial_input: true)
+      raise SmarterJSON::EncodingError, "invalid byte sequence in stream" if status == :invalid_byte_sequence
+
+      out
+    end
+
+    # Flush the converter at end of stream. A held incomplete multibyte sequence means the input
+    # was truncated mid-character — surface it the same way an invalid encoding is surfaced.
+    def finish_transcode(conv)
+      return if conv.nil?
+
+      status = conv.primitive_convert("".b, +"")
+      raise SmarterJSON::EncodingError, "invalid byte sequence in stream" unless status == :finished
+    end
+
     def read_chunk(io)
       if io.respond_to?(:readpartial)
         io.readpartial(CHUNK_SIZE)
@@ -466,6 +671,22 @@ module SmarterJSON
 
     def process_string(input, options, &block)
       input = SmarterJSON.send(:normalize_default_encoding, input, options)
+
+      # UTF-16 / UTF-32 / Shift_JIS / ... cannot be byte-scanned for JSON structure. Parse
+      # a UTF-8 copy and emit each document's strings back in the encoding the bytes arrived
+      # in — the caller always gets values in the encoding they handed us, never UTF-8.
+      if (target_enc = SmarterJSON.send(:unscannable_encoding, input))
+        target_enc = SmarterJSON.send(:concrete_unicode_encoding, input, target_enc) # avoid per-string BOMs
+        opts = options.merge(encoding: nil) # the working copy is UTF-8; don't re-label it downstream
+        utf8 = SmarterJSON.send(:to_utf8_copy, input) # invalid bytes -> SmarterJSON::EncodingError
+        replace = options[:replace_char]
+        if block
+          return process_string(utf8, opts) { |doc| block.call(SmarterJSON.send(:deep_encode, doc, target_enc, replace)) }
+        end
+
+        return process_string(utf8, opts).map { |doc| SmarterJSON.send(:deep_encode, doc, target_enc, replace) }
+      end
+
       return SmarterJSON.send(:process_content, input, options, &block) unless input.valid_encoding?
 
       # Recovery is REACTIVE: parse first, and only fall back to wrapper extraction when

data/lib/smarter_json/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SmarterJSON
-  VERSION = "1.2.2"
+  VERSION = "1.2.4"
 end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: smarter_json
 version: !ruby/object:Gem::Version
-  version: 1.2.2
+  version: 1.2.4
 platform: ruby
 authors:
 - Tilo Sloboda
 bindir: exe
 cert_chain: []
-date: 2026-06-19 00:00:00.000000000 Z
+date: 2026-07-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bigdecimal
@@ -53,6 +53,7 @@ files:
 - docs/basic_write_api.md
 - docs/examples.md
 - docs/options.md
+- ext/smarter_json/cpu_flags.rb
 - ext/smarter_json/extconf.rb
 - ext/smarter_json/smarter_json.c
 - ext/smarter_json/smarter_json.h
@@ -91,7 +92,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.6.9
+rubygems_version: 4.0.15
 specification_version: 4
 summary: A lenient, fast JSON processor for Ruby — reads strict JSON, NDJSON, JSONL,
   JSON5, HJSON, and the messy JSON humans and LLMs actually write.