smarter_json 0.7.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 00a4ada7970ab645761573bd3b9704effda072064102ead760124716f0896289
-  data.tar.gz: a7232f3d8d06d3b7d770645b28a247774c83b8ac50603e4b7611006e852480d4
+  metadata.gz: '06668bbb40626009794f8e5387fb13ae1a31346a07200d2825fde4872904bd68'
+  data.tar.gz: 0db0d42bfd2e85a1af4b897990a290e25e4fbe0afd9b4e25e99d9520667de2c3
 SHA512:
-  metadata.gz: a4b4b8737f10ebf09408014419a9d5cf2203b706766ea5acc358ad595c5cc34104b3f658f2ca6cf6b130124c514d89d035dea2a0ea16e619ffaec7da16da4a23
-  data.tar.gz: a317f74e3399d6b95327fb10a7b65d2ca0f07062d9097303ab32aefb6721a46a78784addd0a937c0c8f433c4acb4e7758af3986fa741bd0d5b816141ee101fd4
+  metadata.gz: 766cd9c5865d7218f79db57ec538bb1d34355c93012e38820544a41bbedbbca94a4a8fce05982f5f2a0e48c5670a6d3f4336eb1a22361fc29b34855d913fc564
+  data.tar.gz: f57396ef1a2ac4e48e06dad94122b8159a3b17c80f6d679e1bb59f85874578ea444592a828c3eb324c8d84ff57c255df0bc99af2aeb5603261aa19d26abc88c2

data/CHANGELOG.md

@@ -3,7 +3,17 @@
 
 > 🚧 Getting ready for the 1.0.0 release - sorry for the interface changes - thank you for your patience! 🚧
 
-## 0.7.0 (2026-06-02)
+## 0.8.0 (2026-06-03)
+- **Robustness** against LLM-generated / wrapped JSON:
+  - strips markdown code fences (```json / ```)
+  - ignores obvious prefix / suffix prose around a payload
+  - unwraps `<json>...</json>` and `BEGIN_JSON ... END_JSON`
+  - preserves multiple recovered payloads as an `Array`
+  - supports pretty-printed multi-line document framing on IO / block input
+  - **Warnings** now cover wrapper recovery too (`:code_fence_stripped`, `:prefix_text_ignored`, `:suffix_text_ignored`, `:wrapper_tag_stripped`)
+  - **No truncation recovery**: truncated / unterminated input still raises `SmarterJSON::ParseError`
+
+## 0.7.0 (2026-06-03)
 - **Breaking:** replaced the `warnings:` option (and its `[result, warnings]` tuple return) with an `on_warning:` callable. Pass `on_warning: ->(w) { ... }` to be handed each `SmarterJSON::Warning` as the parser applies a lenient fix; `process` / `process_file` now always return the bare value (nil / value / Array) on every path. Unlike the tuple, this also fires on the streaming block form. The default (no handler) records nothing and costs nothing.
 
 ## 0.6.0 (2026-06-02)

data/README.md

@@ -16,13 +16,14 @@ Three things set it apart:
 
 1. **One parser, no modes, no flags.** There is no `dialect:` option and no "strict mode" — `SmarterJSON.process(input)` accepts the whole superset, and strict JSON is simply the narrowest case. You don't configure the parser to match your input; it adapts to whatever you give it.
 
-2. **It parses multi-document input automatically — a distinguishing feature.** `SmarterJSON.process` handles NDJSON / JSONL / concatenated JSON with **no block and no special method**: one document returns its value, several documents return an `Array`, empty input returns `nil`. **Only SmarterJSON parses multi-document input via plain `process` — Oj and the stdlib `json` library raise without a block.** For input larger than memory, pass a block to stream one document at a time.
+2. **It parses multi-document input automatically — a distinguishing feature.** `SmarterJSON.process` handles NDJSON / JSONL / concatenated JSON with **no block and no special method**: one document returns its value, several documents return an `Array`, empty input returns `nil`. The same rule applies when wrapper noise is stripped and several payloads are recovered from one blob. **Only SmarterJSON parses multi-document input via plain `process` — Oj and the stdlib `json` library raise without a block.** Pass a block to iterate the recovered documents one at a time.
 
 3. **It's fast.** A C extension (with a pure-Ruby fallback that runs everywhere) puts it ahead of Oj on nearly every file we benchmark, and competitive with the stdlib `json` C parser — the fastest general-purpose Ruby JSON parser.
 
 ## What it accepts, beyond strict JSON
 
 - `//`, `/* … */`, and `#` comments (a `#`/`//` only starts a comment when preceded by whitespace, so `url: http://x.com` parses as a string, not a truncated value)
+- Markdown-wrapped / chatty blobs around the payload: strips ```` ```json ```` / ```` ``` ```` fences, ignores obvious prose before/after the payload, unwraps `<json>...</json>` and `BEGIN_JSON ... END_JSON`, and preserves multiple recovered payloads as an Array
 - Trailing commas; unquoted keys (`{host: localhost}`); single-quoted, triple-quoted (`'''…'''`), and quoteless string values
 - Implicit root object — a config file that starts with `key: value`, no outer `{}`
 - `NaN`, `Infinity`, hex (`0xFF`), leading `+` / `.`, underscores in numbers (`1_000_000`)
@@ -67,9 +68,14 @@ SmarterJSON.process(%({"id":1}\n{"id":2}\n{"id":3}))   # => [{"id"=>1}, {"id"=>2
 SmarterJSON.process('{"id":1}')                         # => {"id"=>1}   (one document → the value itself)
 SmarterJSON.process("")                                 # => nil          (zero documents)
 
-# For input larger than memory, stream one document at a time with a block
+# Iterate one recovered document at a time with a block
 # (process and process_file both forward the block):
 SmarterJSON.process_file("events.ndjson") { |event| EventJob.perform_async(event) }
+
+# Wrapper noise is stripped automatically:
+SmarterJSON.process("Here is the JSON:\n\n```json\n{\"a\":1}\n```\n")  # => {"a"=>1}
+SmarterJSON.process("<json>{\"a\":1}</json>")                          # => {"a"=>1}
+SmarterJSON.process("first:\n{\"a\":1}\nsecond:\n{\"b\":2}")      # => [{"a"=>1}, {"b"=>2}]
 ```
 
 ### Options
@@ -85,7 +91,7 @@ SmarterJSON.process_file("events.ndjson") { |event| EventJob.perform_async(event
 
 ### Warnings (`on_warning`)
 
-When the parser quietly fixes something lenient — collapses an empty comma slot, reads a key with no value as `null`, drops a duplicate key — it can tell you, without changing what `process` returns. Pass a callable as `on_warning:`; it is invoked once per fix with a `SmarterJSON::Warning` (`type`, `message`, `line`, `col`). It fires on every path, including the streaming block form. With no handler (the default) nothing is recorded and there is zero overhead.
+When the parser quietly fixes something lenient — collapses an empty comma slot, reads a key with no value as `null`, drops a duplicate key, strips code fences, ignores wrapper prose, unwraps wrapper tags — it can tell you, without changing what `process` returns. Pass a callable as `on_warning:`; it is invoked once per fix with a `SmarterJSON::Warning` (`type`, `message`, `line`, `col`). It fires on every path, including the streaming block form. With no handler (the default) nothing is recorded and there is zero overhead.
 
 ```ruby
 # Collect them all:
@@ -106,7 +112,7 @@ Benchmarks: p10 of 40 runs, Apple M1 Max, Ruby 3.4.7, on the standard JSON corpu
 
 **Two notes on fair comparison:**
 
-- **NDJSON:** on multi-document files, **only SmarterJSON parses the input via plain `process`** — Oj and `json` raise without a block, so their cells are `N/A`. That `N/A` reflects real default behavior, not a measurement gap. Plain `process` collects every document into an Array at ~270 MB/s; the streaming block form runs faster (~440 MB/s) because it doesn't hold all documents in memory at once — use it for input larger than RAM.
+- **NDJSON:** on multi-document files, **only SmarterJSON parses the input via plain `process`** — Oj and `json` raise without a block, so their cells are `N/A`. That `N/A` reflects real default behavior, not a measurement gap. Plain `process` collects every document into an Array at ~270 MB/s; the block form yields each recovered document instead of returning the collected Array.
 - **High-precision decimals (e.g. `canada.json`):** SmarterJSON's default `:auto` mode preserves high-precision numbers as `BigDecimal` (matching Oj's default), which is intrinsically slower than `Float`. Against `Float`-producing parsers it looks slower on such files; pass `bigdecimal_load: :float` to compare like-for-like (it then runs much faster). Against the equivalent `BigDecimal`-producing Oj mode, SmarterJSON is faster.
 
 ## Encoding

data/docs/_introduction.md

@@ -11,7 +11,7 @@
 
 # SmarterJSON Introduction
 
-`smarter_json` is a fast, lenient JSON parser and writer for Ruby. It reads strict JSON, JSON5, HJSON-style config, newline-delimited JSON (NDJSON / JSONL), and the messy JSON-ish input humans actually paste — and in benchmarks it matches or beats Oj on nearly every file. It is opinionated: it optimizes for getting your data out, not for policing the JSON spec. Where other parsers stop at the first deviation, SmarterJSON keeps going.
+`smarter_json` is a fast, lenient JSON parser and writer for Ruby. It reads strict JSON, JSON5, HJSON-style config, newline-delimited JSON (NDJSON / JSONL), markdown-wrapped / chatty blobs around a JSON payload, and the messy JSON-ish input humans actually paste — and in benchmarks it matches or beats Oj on nearly every file. It is opinionated: it optimizes for getting your data out, not for policing the JSON spec. Where other parsers stop at the first deviation, SmarterJSON keeps going.
 
 ## Why another JSON library?
 
@@ -21,7 +21,7 @@ Most JSON parsers reject anything that isn't perfectly strict JSON, and they mak
 
 * **One reader, no modes, no flags.** There is no `dialect:` option and no "strict mode" — `SmarterJSON.process(input)` accepts the whole superset, and strict JSON is simply the narrowest case. You don't configure the reader to match your input; it adapts to whatever you give it.
 
-* **It reads multi-document input automatically — a distinguishing feature.** `SmarterJSON.process` handles NDJSON / JSONL / concatenated JSON with **no block and no special method**: zero documents returns `nil`, one document returns its value, two or more return an `Array`. **Only SmarterJSON reads multi-document input via plain `process` — Oj and the stdlib `json` library raise without a block.** For input larger than memory, pass a block to stream one document at a time. See [The Basic Read API](./basic_read_api.md).
+* **It reads multi-document input automatically — a distinguishing feature.** `SmarterJSON.process` handles NDJSON / JSONL / concatenated JSON with **no block and no special method**: zero documents returns `nil`, one document returns its value, two or more return an `Array`. The same rule applies when wrapper noise is stripped and several payloads are recovered from one blob. **Only SmarterJSON reads multi-document input via plain `process` — Oj and the stdlib `json` library raise without a block.** Pass a block to iterate the recovered documents one at a time. See [The Basic Read API](./basic_read_api.md).
 
 * **It's fast.** A C extension (with a pure-Ruby fallback that runs everywhere) puts it ahead of Oj on nearly every file we benchmark, and competitive with the stdlib `json` C parser. Floats are parsed with Ryū (correctly rounded, single-pass), so number-heavy data is fast and bit-exact.
 

data/docs/basic_read_api.md

@@ -22,7 +22,7 @@ SmarterJSON.process('{"a": 1, "b": [2, 3]}')          # => {"a"=>1, "b"=>[2, 3]}
 SmarterJSON.process("host: localhost\nport: 5432")     # => {"host"=>"localhost", "port"=>5432}  (no braces needed)
 ```
 
-`process` is polymorphic: its first argument is **either a String of JSON content or an IO to read from**. A String is always treated as content, never as a filename — use `process_file` for paths.
+`process` is polymorphic: its first argument is **either a String of JSON content or an IO to read from**. A String is always treated as content, never as a filename — use `process_file` for paths. When the input wraps the payload in obvious markdown / prose / tags, `process` strips that wrapper first and then parses the recovered payload(s).
 
 ```ruby
 SmarterJSON.process(io)         # an open IO (File, StringIO, socket, …) — reads it and parses
@@ -39,7 +39,7 @@ SmarterJSON.process('{"id":1}')                          # => {"id"=>1}    (one
 SmarterJSON.process(%({"id":1}\n{"id":2}\n{"id":3}))     # => [{"id"=>1}, {"id"=>2}, {"id"=>3}]  (two or more → an Array)
 ```
 
-Documents are separated by whitespace, newlines, or simple concatenation — **not** by commas (a comma between top-level documents would be read as an implicit root array, which is not supported). Only SmarterJSON reads this via plain `process`: Oj and the stdlib `json` library raise without a block.
+Documents are separated by whitespace, newlines, or simple concatenation — **not** by commas (a comma between top-level documents would be read as an implicit root array, which is not supported). If wrapper noise is stripped and several payloads are recovered, they are returned by the same rule: one payload → its value, several → an `Array`. Only SmarterJSON reads this via plain `process`: Oj and the stdlib `json` library raise without a block.
 
 ## `SmarterJSON.process_file` — read a file by path
 
@@ -49,19 +49,16 @@ SmarterJSON.process_file("config.json5")     # read the file, then parse — sam
 
 `process_file` opens the file, reads it with the labeled [`encoding:`](./options.md) (default `"UTF-8"`, no transcoding pass), and parses it.
 
-## Streaming with a block (bounded memory)
+## Streaming with a block
 
-For input larger than memory, pass a block. Each top-level document is yielded as it is read, and the method returns `nil` (it never collects the documents into an Array). Both `process` and `process_file` forward the block.
+Pass a block to have each recovered top-level document yielded one at a time; the method returns `nil` instead of collecting the documents into an Array. Both `process` and `process_file` forward the block.
 
 ```ruby
-# Stream straight from disk, one document at a time — the whole file is never loaded:
 SmarterJSON.process_file("events.ndjson") { |event| EventJob.perform_async(event) }
-
-# Same for an IO:
 SmarterJSON.process(io) { |doc| handle(doc) }
 ```
 
-The streaming path reads the input as newline-delimited documents (NDJSON / JSONL), one document per line. A single document that spans multiple lines is not supported by the streaming path — read it without a block instead.
+The streaming path now frames whole top-level documents, not just one line at a time. That means NDJSON / JSONL still work, but pretty-printed multi-line objects and arrays work too, as do mixed `\n` / `\r\n` / `\r` line endings and comment-only separators between documents.
 
 ## The C extension and the pure-Ruby fallback
 
@@ -79,7 +76,7 @@ warns.map(&:type) # => [:empty_slot]
 warns.first.to_s  # => "extra comma, collapsed an empty slot at line 1, col 4"
 ```
 
-Each warning is a `SmarterJSON::Warning` with `type`, `message`, `line`, and `col`. The types are `:empty_slot` (a collapsed empty comma slot), `:empty_value` (a key with no value, read as `null`), and `:duplicate_key` (a repeated key that was dropped). Clean input never invokes the handler. It fires on every path — including the streaming block form — and works the same on the C and pure-Ruby paths. See [Configuration Options](./options.md).
+Each warning is a `SmarterJSON::Warning` with `type`, `message`, `line`, and `col`. The types are `:empty_slot` (a collapsed empty comma slot), `:empty_value` (a key with no value, read as `null`), `:duplicate_key` (a repeated key that was dropped), plus wrapper-recovery warnings such as `:code_fence_stripped`, `:prefix_text_ignored`, `:suffix_text_ignored`, and `:wrapper_tag_stripped`. Clean input never invokes the handler. It fires on every path — including the streaming block form — and works the same on the C and pure-Ruby paths. See [Configuration Options](./options.md).
 
 ---------------
 

data/docs/examples.md

@@ -24,9 +24,10 @@
 7. [Duplicate Keys](#example-7-duplicate-keys)
 8. [High-Precision Numbers: BigDecimal vs Float](#example-8-high-precision-numbers-bigdecimal-vs-float)
 9. [Lenient Input: Comments, Trailing Commas, Unquoted Keys](#example-9-lenient-input-comments-trailing-commas-unquoted-keys)
-10. [Write JSON](#example-10-write-json)
-11. [Write NDJSON](#example-11-write-ndjson)
-12. [Round-Trip Read and Write](#example-12-round-trip-read-and-write)
+10. [Wrapper Noise Around a Payload](#example-10-wrapper-noise-around-a-payload)
+11. [Write JSON](#example-11-write-json)
+12. [Write NDJSON](#example-12-write-ndjson)
+13. [Round-Trip Read and Write](#example-13-round-trip-read-and-write)
 
 ---
 
@@ -64,9 +65,9 @@ SmarterJSON.process('{"id":1}')                          # => {"id"=>1}
 SmarterJSON.process("")                                  # => nil
 ```
 
-### Example 5: Streaming a Large File with a Block
+### Example 5: Iterate Documents with a Block
 
-For input larger than memory, pass a block. Each document is yielded as it is read; the whole file is never loaded:
+Pass a block to receive each recovered document one at a time:
 
 ```ruby
 SmarterJSON.process_file("events.ndjson") { |event| EventJob.perform_async(event) }
@@ -113,14 +114,41 @@ JSON
 
 A `#`/`//` only starts a comment when preceded by whitespace, so `http://example.com` stays a string rather than being truncated.
 
-### Example 10: Write JSON
+### Example 10: Wrapper Noise Around a Payload
+
+```ruby
+SmarterJSON.process(<<~TEXT)
+  Here is the JSON:
+
+  ```json
+  {
+    "a": 1
+  }
+  ```
+TEXT
+# => {"a"=>1}
+
+SmarterJSON.process("<json>{\"a\":1}</json>")
+# => {"a"=>1}
+
+SmarterJSON.process(<<~TEXT)
+  first:
+  {"a":1}
+
+  second:
+  {"b":2}
+TEXT
+# => [{"a"=>1}, {"b"=>2}]
+```
+
+### Example 11: Write JSON
 
 ```ruby
 SmarterJSON.generate({ "a" => 1, "b" => [2, 3] })   # => '{"a":1,"b":[2,3]}'
 SmarterJSON.generate([1, 2, 3])                       # => '[1,2,3]'
 ```
 
-### Example 11: Write NDJSON
+### Example 12: Write NDJSON
 
 An Array writes one element per line:
 
@@ -128,7 +156,7 @@ An Array writes one element per line:
 SmarterJSON.generate([{ "id" => 1 }, { "id" => 2 }], format: :ndjson)   # => "{\"id\":1}\n{\"id\":2}\n"
 ```
 
-### Example 12: Round-Trip Read and Write
+### Example 13: Round-Trip Read and Write
 
 ```ruby
 obj = { "a" => 1, "b" => [2, "three", nil, true] }

data/docs/options.md

@@ -43,7 +43,7 @@ warns.map(&:type)              # => [:empty_slot]
 warns.first.to_s               # => "extra comma, collapsed an empty slot at line 1, col 4"
 ```
 
-The warning types are `:empty_slot` (a collapsed empty comma slot, e.g. `[1,,2]`), `:empty_value` (a key with no value, read as `null`, e.g. `{a:}`), and `:duplicate_key` (a repeated key that was dropped). Clean input never invokes the handler. Warnings work on both the C and pure-Ruby paths, so `acceleration:` doesn't change them.
+The warning types are `:empty_slot` (a collapsed empty comma slot, e.g. `[1,,2]`), `:empty_value` (a key with no value, read as `null`, e.g. `{a:}`), and `:duplicate_key` (a repeated key that was dropped), plus wrapper-recovery warnings such as `:code_fence_stripped`, `:prefix_text_ignored`, `:suffix_text_ignored`, and `:wrapper_tag_stripped`. Clean input never invokes the handler. Warnings work on both the C and pure-Ruby paths, so `acceleration:` doesn't change them.
 
 ### A note on `:encoding`
 

data/lib/smarter_json/parser.rb

@@ -22,9 +22,9 @@ module SmarterJSON
   # stream as newline-delimited documents (NDJSON / JSONL), one per line.
   def process(input, options = {}, &block)
     if input.is_a?(String)
-      process_content(input, options, &block)
+      Recovery.process_string(input, options, &block)
     elsif input.respond_to?(:read)
-      block ? stream_io(input, options, &block) : process_content(input.read, options)
+      block ? stream_io(input, options, &block) : process(input.read, options)
     else
       raise ArgumentError, "SmarterJSON.process expects a String or an IO, got #{input.class}"
     end
@@ -43,7 +43,7 @@ module SmarterJSON
     if block
       File.open(path, "r:#{encoding}") { |io| stream_io(io, options, &block) }
     else
-      process_content(File.read(path, encoding: encoding), options)
+      process(File.read(path, encoding: encoding), options)
     end
   end
 
@@ -67,12 +67,230 @@ module SmarterJSON
   # each — bounded memory. Newline-delimited (NDJSON / JSONL); a single document
   # spanning multiple lines is not supported by the streaming path.
   def stream_io(io, options, &block)
-    io.each_line("\n") { |line| process_content(line, options, &block) }
-    nil
+    Recovery.process_string(io.read, options, &block)
   end
 
   private_class_method :process_content, :stream_io
 
+  module Recovery
+    module_function
+
+    def process_string(input, options, &block)
+      return SmarterJSON.send(:process_content, input, options, &block) unless input.valid_encoding?
+
+      if wrapper_hint?(input)
+        payloads = extract_payloads(input, options)
+        return replay_payloads(payloads, options, &block) unless payloads.empty?
+      end
+
+      SmarterJSON.send(:process_content, input, options, &block)
+    rescue ParseError => e
+      raise if e.is_a?(EncodingError)
+
+      payloads = extract_payloads(input, options)
+      return replay_payloads(payloads, options, &block) unless payloads.empty?
+
+      raise
+    end
+
+    def wrapper_hint?(input)
+      return false unless input.valid_encoding?
+
+      input.match?(/```|<json\b|BEGIN_JSON\b/i) || input.match?(/\A[[:space:]]*(?:JSON|Final answer)[[:space:]]*:/i)
+    end
+
+    def replay_payloads(payloads, options, &block)
+      handler = options[:on_warning]
+      emit_wrapper_warnings(payloads, handler)
+
+      results = payloads.map do |payload|
+        SmarterJSON.send(:process_content, payload[:slice], options)
+      end
+
+      return results.each(&block).then { nil } if block_given?
+      return nil if results.empty?
+      return results.first if results.length == 1
+
+      results
+    end
+
+    def emit_wrapper_warnings(payloads, handler)
+      return unless handler
+
+      meta = payloads.first[:meta]
+      warn(handler, :prefix_text_ignored, "ignored non-JSON text before the payload", *meta[:first_pos]) if meta[:prefix]
+      warn(handler, :code_fence_stripped, "stripped markdown code fences around the payload", *meta[:first_pos]) if meta[:fence]
+      warn(handler, :wrapper_tag_stripped, "stripped wrapper tags around the payload", *meta[:first_pos]) if meta[:wrapper]
+      warn(handler, :suffix_text_ignored, "ignored non-JSON text after the payload", *meta[:last_pos]) if meta[:suffix]
+    end
+
+    def extract_payloads(input, options)
+      payloads = candidate_ranges(input).filter_map do |range|
+        slice = input.byteslice(range.begin, range.end - range.begin)
+        begin
+          SmarterJSON.send(:process_content, slice, options.merge(on_warning: nil))
+          { slice: slice, range: range }
+        rescue ParseError
+          nil
+        end
+      end
+      meta = wrapper_meta(input, payloads.map { |p| p[:range] })
+      payloads.each { |payload| payload[:meta] = meta }
+      payloads
+    end
+
+    def wrapper_meta(input, ranges)
+      return { prefix: false, suffix: false, fence: false, wrapper: false } if ranges.empty?
+
+      first = ranges.first
+      last = ranges.last
+      prefix = input.byteslice(0, first.begin)
+      suffix = input.byteslice(last.end, input.bytesize - last.end)
+      {
+        prefix: substantive_text?(prefix),
+        suffix: substantive_text?(suffix),
+        fence: input.match?(/```/),
+        wrapper: input.match?(/<json\b|BEGIN_JSON\b/i),
+        first_pos: line_col_for(input, first.begin),
+        last_pos: line_col_for(input, last.begin)
+      }
+    end
+
+    def line_col_for(input, offset)
+      line = 1
+      col = 1
+      i = 0
+      while i < offset
+        b = input.getbyte(i)
+        break if b.nil?
+
+        if b == 0x0A
+          line += 1
+          col = 1
+          i += 1
+        elsif b == 0x0D
+          line += 1
+          col = 1
+          i += 1
+          i += 1 if i < offset && input.getbyte(i) == 0x0A
+        else
+          col += 1
+          i += 1
+        end
+      end
+      [line, col]
+    end
+
+    def substantive_text?(text)
+      return false if text.nil? || text.empty?
+
+      stripped = text.dup
+      stripped.gsub!(%r{/\*.*?\*/}m, "")
+      stripped.gsub!(/^\s*(?:#|\/\/).*$/, "")
+      !stripped.strip.empty? && !stripped.strip.match?(/\A(?:```[a-zA-Z0-9_-]*)?\z/) && !stripped.strip.match?(/\A(?:<\/?json>|BEGIN_JSON|END_JSON)\z/i)
+    end
+
+    def warn(handler, type, message, line, col)
+      handler.call(Warning.new(type, message, line, col))
+    end
+
+    def candidate_ranges(input)
+      ranges = []
+      stack = []
+      start_pos = nil
+      i = 0
+      mode = nil
+      while i < input.bytesize
+        b = input.getbyte(i)
+        if mode == :double
+          if b == 0x5C
+            i += 2
+            next
+          elsif b == 0x22
+            mode = nil
+          end
+          i += 1
+          next
+        elsif mode == :single
+          if b == 0x5C
+            i += 2
+            next
+          elsif b == 0x27
+            mode = nil
+          end
+          i += 1
+          next
+        elsif mode == :triple
+          if input.byteslice(i, 3) == "'''"
+            mode = nil
+            i += 3
+          else
+            i += 1
+          end
+          next
+        elsif mode == :line_comment
+          if [0x0A, 0x0D].include?(b)
+            mode = nil
+          else
+            i += 1
+            next
+          end
+        elsif mode == :block_comment
+          if input.byteslice(i, 2) == "*/"
+            mode = nil
+            i += 2
+          else
+            i += 1
+          end
+          next
+        else
+          if input.byteslice(i, 2) == "//"
+            mode = :line_comment
+            i += 2
+            next
+          elsif input.byteslice(i, 2) == "/*"
+            mode = :block_comment
+            i += 2
+            next
+          elsif b == 0x23
+            mode = :line_comment
+            i += 1
+            next
+          elsif b == 0x22
+            mode = :double
+            i += 1
+            next
+          elsif input.byteslice(i, 3) == "'''"
+            mode = :triple
+            i += 3
+            next
+          elsif b == 0x27
+            mode = :single
+            i += 1
+            next
+          elsif [0x7B, 0x5B].include?(b)
+            start_pos = i if stack.empty?
+            stack << b
+          elsif b == 0x7D
+            stack.pop if stack.last == 0x7B
+            if stack.empty? && start_pos
+              ranges << (start_pos...(i + 1))
+              start_pos = nil
+            end
+          elsif b == 0x5D
+            stack.pop if stack.last == 0x5B
+            if stack.empty? && start_pos
+              ranges << (start_pos...(i + 1))
+              start_pos = nil
+            end
+          end
+        end
+        i += 1
+      end
+      ranges
+    end
+  end
+
   # Hand-rolled FSM single-pass parser.
   # Layer 1: strict JSON (RFC 8259).
   # Layer 2: JSON5 additions — line/block comments, trailing comma,

data/lib/smarter_json/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SmarterJSON
-  VERSION = "0.7.0"
+  VERSION = "0.8.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: smarter_json
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.8.0
 platform: ruby
 authors:
 - Tilo Sloboda