smarter_json 0.6.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 65b2a8f15607b861034e600fd42233328803ebbcb652e8cb8b162811e24779e2
-  data.tar.gz: 191467cd1d29030d054c1cdaabb444523af54fb71486721182ddcdf9a4781d92
+  metadata.gz: '06668bbb40626009794f8e5387fb13ae1a31346a07200d2825fde4872904bd68'
+  data.tar.gz: 0db0d42bfd2e85a1af4b897990a290e25e4fbe0afd9b4e25e99d9520667de2c3
 SHA512:
-  metadata.gz: c26d4ffb0789fbe7303b80d80877f02164f79f7fe2ae9eff0e17de90df4ba8da19568c7a1fa5d10d5d324c8f4a841cd8ad0917f5ab6a5f381368404f56a1ec5c
-  data.tar.gz: 6c3fbbf21ec9f9c6300516268ad4e7741aeaab8a6732ab138497f6af5b02afbfd944fdce6d4c7095f699469b77fd2fa2cf5e65ea58712ad55da0e5c745fd6551
+  metadata.gz: 766cd9c5865d7218f79db57ec538bb1d34355c93012e38820544a41bbedbbca94a4a8fce05982f5f2a0e48c5670a6d3f4336eb1a22361fc29b34855d913fc564
+  data.tar.gz: f57396ef1a2ac4e48e06dad94122b8159a3b17c80f6d679e1bb59f85874578ea444592a828c3eb324c8d84ff57c255df0bc99af2aeb5603261aa19d26abc88c2

data/CHANGELOG.md

@@ -1,6 +1,21 @@
 
 # SmarterJSON Change Log
 
+> 🚧 Getting ready for the 1.0.0 release - sorry for the interface changes - thank you for your patience! 🚧
+
+## 0.8.0 (2026-06-03)
+- **Robustness** against LLM-generated / wrapped JSON:
+  - strips markdown code fences (```json / ```)
+  - ignores obvious prefix / suffix prose around a payload
+  - unwraps `<json>...</json>` and `BEGIN_JSON ... END_JSON`
+  - preserves multiple recovered payloads as an `Array`
+  - supports pretty-printed multi-line document framing on IO / block input
+  - **Warnings** now cover wrapper recovery too (`:code_fence_stripped`, `:prefix_text_ignored`, `:suffix_text_ignored`, `:wrapper_tag_stripped`)
+  - **No truncation recovery**: truncated / unterminated input still raises `SmarterJSON::ParseError`
+
+## 0.7.0 (2026-06-03)
+- **Breaking:** replaced the `warnings:` option (and its `[result, warnings]` tuple return) with an `on_warning:` callable. Pass `on_warning: ->(w) { ... }` to be handed each `SmarterJSON::Warning` as the parser applies a lenient fix; `process` / `process_file` now always return the bare value (nil / value / Array) on every path. Unlike the tuple, this also fires on the streaming block form. The default (no handler) records nothing and costs nothing.
+
 ## 0.6.0 (2026-06-02)
 - Lenient comma handling: empty slots around / between commas are collapsed (`[1,,2]` → `[1,2]`, `[,1,]` → `[1]`, `{a:1,,b:2}` → `{a:1,b:2}`), on both the C and Ruby paths. No null is inserted for an empty slot.
 - A key with a colon but no value reads as null: `{a:}` → `{"a"=>nil}` (both paths).
@@ -9,12 +24,12 @@
 - Fixed a pure-Ruby bug where a `\u` escape whose next bytes split a multibyte character leaked `ArgumentError`; it now raises `SmarterJSON::ParseError`.
 - Added a property/fuzz test suite that checks C/Ruby parity and round-tripping on generated, mutated, and random input.
 
-## 0.5.2 (2026-06-01)
+## 0.5.2 (2026-06-01) yanked
 - `generate` now supports pretty-printing via the `indent:` option (spaces per nesting level; default `0` = compact). Empty objects/arrays stay inline; `indent:` combined with `format: :ndjson` raises `ArgumentError`.
 - `generate` adds `sort_keys:` (emit object keys in sorted order), `ascii_only:` (escape non-ASCII as `\uXXXX`, astral chars as surrogate pairs), and `script_safe:` (escape `</` and U+2028/U+2029 for safe embedding in an HTML `<script>` tag).
 - `generate` adds opt-in `coerce:` — when `true`, a value that isn't natively supported (e.g. `Time`, `Date`, app objects) is converted via its own `as_json` (result re-emitted) or `to_json` (spliced); strict-by-default still raises `GenerateError`.
 
-## 0.5.1 (2026-06-01)
+## 0.5.1 (2026-06-01) yanked
 - Unified the error classes under a single `SmarterJSON::Error` base: `ParseError` and `EncodingError` now inherit from it, and `generate` raises a new `GenerateError`. `rescue SmarterJSON::Error` now catches everything the gem raises.
 - Added a CI test matrix (Ruby 2.6–4.0 + head, on Ubuntu and macOS).
 - Fixed the C extension build on Ruby 2.6 (declare `rb_hash_bulk_insert`, which 2.6 exports but does not declare in its headers); set the minimum Ruby to 2.6.

data/README.md

@@ -16,13 +16,14 @@ Three things set it apart:
 
 1. **One parser, no modes, no flags.** There is no `dialect:` option and no "strict mode" — `SmarterJSON.process(input)` accepts the whole superset, and strict JSON is simply the narrowest case. You don't configure the parser to match your input; it adapts to whatever you give it.
 
-2. **It parses multi-document input automatically — a distinguishing feature.** `SmarterJSON.process` handles NDJSON / JSONL / concatenated JSON with **no block and no special method**: one document returns its value, several documents return an `Array`, empty input returns `nil`. **Only SmarterJSON parses multi-document input via plain `process` — Oj and the stdlib `json` library raise without a block.** For input larger than memory, pass a block to stream one document at a time.
+2. **It parses multi-document input automatically — a distinguishing feature.** `SmarterJSON.process` handles NDJSON / JSONL / concatenated JSON with **no block and no special method**: one document returns its value, several documents return an `Array`, empty input returns `nil`. The same rule applies when wrapper noise is stripped and several payloads are recovered from one blob. **Only SmarterJSON parses multi-document input via plain `process` — Oj and the stdlib `json` library raise without a block.** Pass a block to iterate the recovered documents one at a time.
 
 3. **It's fast.** A C extension (with a pure-Ruby fallback that runs everywhere) puts it ahead of Oj on nearly every file we benchmark, and competitive with the stdlib `json` C parser — the fastest general-purpose Ruby JSON parser.
 
 ## What it accepts, beyond strict JSON
 
 - `//`, `/* … */`, and `#` comments (a `#`/`//` only starts a comment when preceded by whitespace, so `url: http://x.com` parses as a string, not a truncated value)
+- Markdown-wrapped / chatty blobs around the payload: strips ```` ```json ```` / ```` ``` ```` fences, ignores obvious prose before/after the payload, unwraps `<json>...</json>` and `BEGIN_JSON ... END_JSON`, and preserves multiple recovered payloads as an Array
 - Trailing commas; unquoted keys (`{host: localhost}`); single-quoted, triple-quoted (`'''…'''`), and quoteless string values
 - Implicit root object — a config file that starts with `key: value`, no outer `{}`
 - `NaN`, `Infinity`, hex (`0xFF`), leading `+` / `.`, underscores in numbers (`1_000_000`)
@@ -67,9 +68,14 @@ SmarterJSON.process(%({"id":1}\n{"id":2}\n{"id":3}))   # => [{"id"=>1}, {"id"=>2
 SmarterJSON.process('{"id":1}')                         # => {"id"=>1}   (one document → the value itself)
 SmarterJSON.process("")                                 # => nil          (zero documents)
 
-# For input larger than memory, stream one document at a time with a block
+# Iterate one recovered document at a time with a block
 # (process and process_file both forward the block):
 SmarterJSON.process_file("events.ndjson") { |event| EventJob.perform_async(event) }
+
+# Wrapper noise is stripped automatically:
+SmarterJSON.process("Here is the JSON:\n\n```json\n{\"a\":1}\n```\n")  # => {"a"=>1}
+SmarterJSON.process("<json>{\"a\":1}</json>")                          # => {"a"=>1}
+SmarterJSON.process("first:\n{\"a\":1}\nsecond:\n{\"b\":2}")      # => [{"a"=>1}, {"b"=>2}]
 ```
 
 ### Options
@@ -81,7 +87,20 @@ SmarterJSON.process_file("events.ndjson") { |event| EventJob.perform_async(event
 | `bigdecimal_load` | `:auto`      | `:auto` keeps high-precision decimals as `BigDecimal`; `:float` forces `Float`; `:bigdecimal` forces `BigDecimal` |
 | `acceleration`    | `true`       | `true` uses the C extension when compiled and loadable; `false` forces pure Ruby (identical results) |
 | `encoding`        | `"UTF-8"`    | labels the input's encoding (no transcoding pass; see below)            |
-| `warnings`        | `false`      | when `true`, return `[result, warnings]` — `warnings` lists the lenient fixes applied (`:empty_slot`, `:empty_value`, `:duplicate_key`) |
+| `on_warning`      | `nil`        | a callable invoked once per lenient fix applied (`:empty_slot`, `:empty_value`, `:duplicate_key`), passed a `SmarterJSON::Warning`; the return value is never changed. See below. |
+
+### Warnings (`on_warning`)
+
+When the parser quietly fixes something lenient — collapses an empty comma slot, reads a key with no value as `null`, drops a duplicate key, strips code fences, ignores wrapper prose, unwraps wrapper tags — it can tell you, without changing what `process` returns. Pass a callable as `on_warning:`; it is invoked once per fix with a `SmarterJSON::Warning` (`type`, `message`, `line`, `col`). It fires on every path, including the streaming block form. With no handler (the default) nothing is recorded and there is zero overhead.
+
+```ruby
+# Collect them all:
+warns = []
+data  = SmarterJSON.process(input, on_warning: ->(w) { warns << w })
+
+# Or route them — log, count, raise:
+SmarterJSON.process(input, on_warning: ->(w) { Rails.logger.warn(w) })
+```
 
 ## Performance
 
@@ -93,7 +112,7 @@ Benchmarks: p10 of 40 runs, Apple M1 Max, Ruby 3.4.7, on the standard JSON corpu
 
 **Two notes on fair comparison:**
 
-- **NDJSON:** on multi-document files, **only SmarterJSON parses the input via plain `process`** — Oj and `json` raise without a block, so their cells are `N/A`. That `N/A` reflects real default behavior, not a measurement gap. Plain `process` collects every document into an Array at ~270 MB/s; the streaming block form runs faster (~440 MB/s) because it doesn't hold all documents in memory at once — use it for input larger than RAM.
+- **NDJSON:** on multi-document files, **only SmarterJSON parses the input via plain `process`** — Oj and `json` raise without a block, so their cells are `N/A`. That `N/A` reflects real default behavior, not a measurement gap. Plain `process` collects every document into an Array at ~270 MB/s; the block form yields each recovered document instead of returning the collected Array.
 - **High-precision decimals (e.g. `canada.json`):** SmarterJSON's default `:auto` mode preserves high-precision numbers as `BigDecimal` (matching Oj's default), which is intrinsically slower than `Float`. Against `Float`-producing parsers it looks slower on such files; pass `bigdecimal_load: :float` to compare like-for-like (it then runs much faster). Against the equivalent `BigDecimal`-producing Oj mode, SmarterJSON is faster.
 
 ## Encoding

data/docs/_introduction.md

@@ -11,7 +11,7 @@
 
 # SmarterJSON Introduction
 
-`smarter_json` is a fast, lenient JSON parser and writer for Ruby. It reads strict JSON, JSON5, HJSON-style config, newline-delimited JSON (NDJSON / JSONL), and the messy JSON-ish input humans actually paste — and in benchmarks it matches or beats Oj on nearly every file. It is opinionated: it optimizes for getting your data out, not for policing the JSON spec. Where other parsers stop at the first deviation, SmarterJSON keeps going.
+`smarter_json` is a fast, lenient JSON parser and writer for Ruby. It reads strict JSON, JSON5, HJSON-style config, newline-delimited JSON (NDJSON / JSONL), markdown-wrapped / chatty blobs around a JSON payload, and the messy JSON-ish input humans actually paste — and in benchmarks it matches or beats Oj on nearly every file. It is opinionated: it optimizes for getting your data out, not for policing the JSON spec. Where other parsers stop at the first deviation, SmarterJSON keeps going.
 
 ## Why another JSON library?
 
@@ -21,7 +21,7 @@ Most JSON parsers reject anything that isn't perfectly strict JSON, and they mak
 
 * **One reader, no modes, no flags.** There is no `dialect:` option and no "strict mode" — `SmarterJSON.process(input)` accepts the whole superset, and strict JSON is simply the narrowest case. You don't configure the reader to match your input; it adapts to whatever you give it.
 
-* **It reads multi-document input automatically — a distinguishing feature.** `SmarterJSON.process` handles NDJSON / JSONL / concatenated JSON with **no block and no special method**: zero documents returns `nil`, one document returns its value, two or more return an `Array`. **Only SmarterJSON reads multi-document input via plain `process` — Oj and the stdlib `json` library raise without a block.** For input larger than memory, pass a block to stream one document at a time. See [The Basic Read API](./basic_read_api.md).
+* **It reads multi-document input automatically — a distinguishing feature.** `SmarterJSON.process` handles NDJSON / JSONL / concatenated JSON with **no block and no special method**: zero documents returns `nil`, one document returns its value, two or more return an `Array`. The same rule applies when wrapper noise is stripped and several payloads are recovered from one blob. **Only SmarterJSON reads multi-document input via plain `process` — Oj and the stdlib `json` library raise without a block.** Pass a block to iterate the recovered documents one at a time. See [The Basic Read API](./basic_read_api.md).
 
 * **It's fast.** A C extension (with a pure-Ruby fallback that runs everywhere) puts it ahead of Oj on nearly every file we benchmark, and competitive with the stdlib `json` C parser. Floats are parsed with Ryū (correctly rounded, single-pass), so number-heavy data is fast and bit-exact.
 

data/docs/basic_read_api.md

@@ -22,7 +22,7 @@ SmarterJSON.process('{"a": 1, "b": [2, 3]}')          # => {"a"=>1, "b"=>[2, 3]}
 SmarterJSON.process("host: localhost\nport: 5432")     # => {"host"=>"localhost", "port"=>5432}  (no braces needed)
 ```
 
-`process` is polymorphic: its first argument is **either a String of JSON content or an IO to read from**. A String is always treated as content, never as a filename — use `process_file` for paths.
+`process` is polymorphic: its first argument is **either a String of JSON content or an IO to read from**. A String is always treated as content, never as a filename — use `process_file` for paths. When the input wraps the payload in obvious markdown / prose / tags, `process` strips that wrapper first and then parses the recovered payload(s).
 
 ```ruby
 SmarterJSON.process(io)         # an open IO (File, StringIO, socket, …) — reads it and parses
@@ -39,7 +39,7 @@ SmarterJSON.process('{"id":1}')                          # => {"id"=>1}    (one
 SmarterJSON.process(%({"id":1}\n{"id":2}\n{"id":3}))     # => [{"id"=>1}, {"id"=>2}, {"id"=>3}]  (two or more → an Array)
 ```
 
-Documents are separated by whitespace, newlines, or simple concatenation — **not** by commas (a comma between top-level documents would be read as an implicit root array, which is not supported). Only SmarterJSON reads this via plain `process`: Oj and the stdlib `json` library raise without a block.
+Documents are separated by whitespace, newlines, or simple concatenation — **not** by commas (a comma between top-level documents would be read as an implicit root array, which is not supported). If wrapper noise is stripped and several payloads are recovered, they are returned by the same rule: one payload → its value, several → an `Array`. Only SmarterJSON reads this via plain `process`: Oj and the stdlib `json` library raise without a block.
 
 ## `SmarterJSON.process_file` — read a file by path
 
@@ -49,36 +49,34 @@ SmarterJSON.process_file("config.json5")     # read the file, then parse — sam
 
 `process_file` opens the file, reads it with the labeled [`encoding:`](./options.md) (default `"UTF-8"`, no transcoding pass), and parses it.
 
-## Streaming with a block (bounded memory)
+## Streaming with a block
 
-For input larger than memory, pass a block. Each top-level document is yielded as it is read, and the method returns `nil` (it never collects the documents into an Array). Both `process` and `process_file` forward the block.
+Pass a block to have each recovered top-level document yielded one at a time; the method returns `nil` instead of collecting the documents into an Array. Both `process` and `process_file` forward the block.
 
 ```ruby
-# Stream straight from disk, one document at a time — the whole file is never loaded:
 SmarterJSON.process_file("events.ndjson") { |event| EventJob.perform_async(event) }
-
-# Same for an IO:
 SmarterJSON.process(io) { |doc| handle(doc) }
 ```
 
-The streaming path reads the input as newline-delimited documents (NDJSON / JSONL), one document per line. A single document that spans multiple lines is not supported by the streaming path — read it without a block instead.
+The streaming path now frames whole top-level documents, not just one line at a time. That means NDJSON / JSONL still work, but pretty-printed multi-line objects and arrays work too, as do mixed `\n` / `\r\n` / `\r` line endings and comment-only separators between documents.
 
 ## The C extension and the pure-Ruby fallback
 
 By default (`acceleration: true`) the C extension is used when it is compiled and loadable (`SmarterJSON::HAS_ACCELERATION` is then `true`); otherwise the pure-Ruby parser runs and produces identical results. Pass `acceleration: false` to force the pure-Ruby path. See [Configuration Options](./options.md).
 
-## Seeing what was fixed: `warnings:`
+## Seeing what was fixed: `on_warning:`
 
-`process` and `process_file` are lenient — they salvage your data rather than reject a whole document over a stray comma. Pass `warnings: true` to also get back a record of what was adjusted, so the leniency is transparent instead of silent. The call then returns `[result, warnings]`:
+`process` and `process_file` are lenient — they salvage your data rather than reject a whole document over a stray comma. Pass an `on_warning:` callable to also get a record of what was adjusted, so the leniency is transparent instead of silent. It is invoked once per fix and never changes the return value:
 
 ```ruby
-result, warnings = SmarterJSON.process("[1,,2]", warnings: true)
-result               # => [1, 2]
-warnings.map(&:type) # => [:empty_slot]
-warnings.first.to_s  # => "extra comma, collapsed an empty slot at line 1, col 4"
+warns = []
+result = SmarterJSON.process("[1,,2]", on_warning: ->(w) { warns << w })
+result            # => [1, 2]
+warns.map(&:type) # => [:empty_slot]
+warns.first.to_s  # => "extra comma, collapsed an empty slot at line 1, col 4"
 ```
 
-Each warning is a `SmarterJSON::Warning` with `type`, `message`, `line`, and `col`. The types are `:empty_slot` (a collapsed empty comma slot), `:empty_value` (a key with no value, read as `null`), and `:duplicate_key` (a repeated key that was dropped). Clean input gives an empty `warnings` array. It works the same on the C and pure-Ruby paths. See [Configuration Options](./options.md).
+Each warning is a `SmarterJSON::Warning` with `type`, `message`, `line`, and `col`. The types are `:empty_slot` (a collapsed empty comma slot), `:empty_value` (a key with no value, read as `null`), `:duplicate_key` (a repeated key that was dropped), plus wrapper-recovery warnings such as `:code_fence_stripped`, `:prefix_text_ignored`, `:suffix_text_ignored`, and `:wrapper_tag_stripped`. Clean input never invokes the handler. It fires on every path — including the streaming block form — and works the same on the C and pure-Ruby paths. See [Configuration Options](./options.md).
 
 ---------------
 

data/docs/examples.md

@@ -24,9 +24,10 @@
 7. [Duplicate Keys](#example-7-duplicate-keys)
 8. [High-Precision Numbers: BigDecimal vs Float](#example-8-high-precision-numbers-bigdecimal-vs-float)
 9. [Lenient Input: Comments, Trailing Commas, Unquoted Keys](#example-9-lenient-input-comments-trailing-commas-unquoted-keys)
-10. [Write JSON](#example-10-write-json)
-11. [Write NDJSON](#example-11-write-ndjson)
-12. [Round-Trip Read and Write](#example-12-round-trip-read-and-write)
+10. [Wrapper Noise Around a Payload](#example-10-wrapper-noise-around-a-payload)
+11. [Write JSON](#example-11-write-json)
+12. [Write NDJSON](#example-12-write-ndjson)
+13. [Round-Trip Read and Write](#example-13-round-trip-read-and-write)
 
 ---
 
@@ -64,9 +65,9 @@ SmarterJSON.process('{"id":1}')                          # => {"id"=>1}
 SmarterJSON.process("")                                  # => nil
 ```
 
-### Example 5: Streaming a Large File with a Block
+### Example 5: Iterate Documents with a Block
 
-For input larger than memory, pass a block. Each document is yielded as it is read; the whole file is never loaded:
+Pass a block to receive each recovered document one at a time:
 
 ```ruby
 SmarterJSON.process_file("events.ndjson") { |event| EventJob.perform_async(event) }
@@ -113,14 +114,41 @@ JSON
 
 A `#`/`//` only starts a comment when preceded by whitespace, so `http://example.com` stays a string rather than being truncated.
 
-### Example 10: Write JSON
+### Example 10: Wrapper Noise Around a Payload
+
+```ruby
+SmarterJSON.process(<<~TEXT)
+  Here is the JSON:
+
+  ```json
+  {
+    "a": 1
+  }
+  ```
+TEXT
+# => {"a"=>1}
+
+SmarterJSON.process("<json>{\"a\":1}</json>")
+# => {"a"=>1}
+
+SmarterJSON.process(<<~TEXT)
+  first:
+  {"a":1}
+
+  second:
+  {"b":2}
+TEXT
+# => [{"a"=>1}, {"b"=>2}]
+```
+
+### Example 11: Write JSON
 
 ```ruby
 SmarterJSON.generate({ "a" => 1, "b" => [2, 3] })   # => '{"a":1,"b":[2,3]}'
 SmarterJSON.generate([1, 2, 3])                       # => '[1,2,3]'
 ```
 
-### Example 11: Write NDJSON
+### Example 12: Write NDJSON
 
 An Array writes one element per line:
 
@@ -128,7 +156,7 @@ An Array writes one element per line:
 SmarterJSON.generate([{ "id" => 1 }, { "id" => 2 }], format: :ndjson)   # => "{\"id\":1}\n{\"id\":2}\n"
 ```
 
-### Example 12: Round-Trip Read and Write
+### Example 13: Round-Trip Read and Write
 
 ```ruby
 obj = { "a" => 1, "b" => [2, "three", nil, true] }

data/docs/options.md

@@ -22,27 +22,28 @@ These options are passed to [`SmarterJSON.process`](./basic_read_api.md) and `Sm
 | `:bigdecimal_load`| `:auto`      | `:auto` keeps high-precision decimals as `BigDecimal` (matches Oj); `:float` forces every number to `Float`; `:bigdecimal` forces every decimal to `BigDecimal`. |
 | `:acceleration`   | `true`       | Use the C extension when it is compiled and loadable; `false` forces the pure-Ruby parser. Both produce identical results. |
 | `:encoding`       | `nil`        | Labels the input's encoding (e.g. `"UTF-8"`). It does **not** trigger a transcoding pass — see below.                  |
-| `:warnings`       | `false`      | When `true`, return `[result, warnings]` instead of just `result` — `warnings` lists the lenient fixes that were applied. See below. |
+| `:on_warning`     | `nil`        | A callable invoked once per lenient fix applied, passed a `SmarterJSON::Warning`. Never changes the return value. See below. |
 
 ```ruby
 SmarterJSON.process('{"a": 1}', symbolize_keys: true)               # => {:a=>1}
 SmarterJSON.process('{"a":1,"a":2}', duplicate_key: :raise)         # raises SmarterJSON::ParseError
 SmarterJSON.process(big_decimal_json, bigdecimal_load: :float)      # every number as Float (fastest)
-SmarterJSON.process("[1,,2]", warnings: true)                       # => [[1, 2], [#<SmarterJSON::Warning ...>]]
+SmarterJSON.process("[1,,2]", on_warning: ->(w) { puts w })         # => [1, 2], and prints the warning
 ```
 
-### A note on `:warnings`
+### A note on `:on_warning`
 
-`smarter_json` is lenient by design — it salvages your data instead of rejecting the whole document over a stray comma. `warnings: true` keeps that, but also hands back a record of what it had to fix, so leniency is transparent rather than silent. The call then returns a two-element `[result, warnings]`; `warnings` is an Array of `SmarterJSON::Warning`, each with `type` (a Symbol), `message`, `line`, and `col`:
+`smarter_json` is lenient by design — it salvages your data instead of rejecting the whole document over a stray comma. `on_warning:` keeps that, but also hands you a record of what it had to fix, so leniency is transparent rather than silent. It takes a callable that the parser invokes once per fix, passing a `SmarterJSON::Warning` (with `type` (a Symbol), `message`, `line`, and `col`). It never changes the return value — `process` still hands back the bare value — and it fires on every path, including the streaming block form. With no handler (the default), nothing is recorded and there is zero overhead.
 
 ```ruby
-result, warnings = SmarterJSON.process("[1,,2]", warnings: true)
+warns = []
+result = SmarterJSON.process("[1,,2]", on_warning: ->(w) { warns << w })
 result                         # => [1, 2]
-warnings.map(&:type)           # => [:empty_slot]
-warnings.first.to_s            # => "extra comma, collapsed an empty slot at line 1, col 4"
+warns.map(&:type)              # => [:empty_slot]
+warns.first.to_s               # => "extra comma, collapsed an empty slot at line 1, col 4"
 ```
 
-The warning types are `:empty_slot` (a collapsed empty comma slot, e.g. `[1,,2]`), `:empty_value` (a key with no value, read as `null`, e.g. `{a:}`), and `:duplicate_key` (a repeated key that was dropped). Clean input returns an empty `warnings` array. Warnings work on both the C and pure-Ruby paths, so `acceleration:` doesn't change them.
+The warning types are `:empty_slot` (a collapsed empty comma slot, e.g. `[1,,2]`), `:empty_value` (a key with no value, read as `null`, e.g. `{a:}`), and `:duplicate_key` (a repeated key that was dropped), plus wrapper-recovery warnings such as `:code_fence_stripped`, `:prefix_text_ignored`, `:suffix_text_ignored`, and `:wrapper_tag_stripped`. Clean input never invokes the handler. Warnings work on both the C and pure-Ruby paths, so `acceleration:` doesn't change them.
 
 ### A note on `:encoding`
 

data/ext/smarter_json/smarter_json.c

@@ -34,6 +34,7 @@ static VALUE cParseError;
 static VALUE cEncodingError;
 static VALUE cWarning;
 static ID    fj_new_id;
+static ID    fj_call_id;    /* cached :call (invoking the on_warning handler) */
 static VALUE fj_sym_empty_slot;
 static VALUE fj_sym_empty_value;
 static VALUE fj_sym_duplicate_key;
@@ -60,8 +61,7 @@ typedef struct {
   int dup_raise;
   int bigdecimal_load;  /* 0 = float, 1 = auto, 2 = bigdecimal */
   fj_kc_slot *kcache;   /* per-parse key cache (NULL when interning unavailable) */
-  int collect_warnings; /* warnings: option — record non-fatal lenient fixes */
-  VALUE warnings;       /* rb_ary of SmarterJSON::Warning when collecting, else Qnil */
+  VALUE on_warning;     /* on_warning: callable invoked per non-fatal lenient fix, else Qnil */
 } fj_state;
 
 /* Line/column at the current byte position, computed lazily (only when raising
@@ -81,14 +81,16 @@ static void fj_line_col(fj_state *st, long *line, long *col) {
   *col = c;
 }
 
-/* Record a non-fatal lenient fix — only when the parse was given warnings: true. */
+/* Report a non-fatal lenient fix to the on_warning callable — a no-op (and builds no
+ * Warning) when no handler was given. The internal Qnil guard is the safety net; the
+ * call sites also guard so the line/col scan is skipped on the fast path. */
 static void fj_warn(fj_state *st, VALUE type_sym, const char *msg) {
   long line, col;
-  if (!st->collect_warnings) return;
+  if (st->on_warning == Qnil) return;
   fj_line_col(st, &line, &col);
-  rb_ary_push(st->warnings,
-              rb_funcall(cWarning, fj_new_id, 4, type_sym,
-                         rb_utf8_str_new_cstr(msg), LONG2NUM(line), LONG2NUM(col)));
+  rb_funcall(st->on_warning, fj_call_id, 1,
+             rb_funcall(cWarning, fj_new_id, 4, type_sym,
+                        rb_utf8_str_new_cstr(msg), LONG2NUM(line), LONG2NUM(col)));
 }
 
 /* 1-based column of the current byte position (bytes since the last line start).
@@ -1161,9 +1163,9 @@ static VALUE fj_build_object(fj_state *st, const VALUE *pairs, long count) {
   long  entries = count / 2, i;
   VALUE hash    = rb_hash_new_capa(entries);
 
-  /* Fast path: bulk insert. Skipped when collecting warnings, which needs the
-   * per-member loop below to report each dropped duplicate key. */
-  if (!st->symbolize_keys && !st->dup_first_wins && !st->collect_warnings) {
+  /* Fast path: bulk insert. Skipped when an on_warning handler is present, which needs
+   * the per-member loop below to report each dropped duplicate key. */
+  if (!st->symbolize_keys && !st->dup_first_wins && st->on_warning == Qnil) {
     rb_hash_bulk_insert(count, pairs, hash);
     if (st->dup_raise && fj_hash_len(hash) < entries) {
       VALUE seen = rb_hash_new_capa(entries);
@@ -1178,7 +1180,7 @@ static VALUE fj_build_object(fj_state *st, const VALUE *pairs, long count) {
 
   for (i = 0; i + 1 < count; i += 2) {
     VALUE k = st->symbolize_keys ? rb_funcall(pairs[i], fj_to_sym_id, 0) : pairs[i];
-    if (st->dup_first_wins || st->dup_raise || st->collect_warnings) {
+    if (st->dup_first_wins || st->dup_raise || st->on_warning != Qnil) {
       if (RTEST(rb_funcall(hash, fj_key_p_id, 1, k))) {
         if (st->dup_raise) fj_error(st, "duplicate key");
         fj_warn(st, fj_sym_duplicate_key, "duplicate key");
@@ -1271,7 +1273,7 @@ static VALUE fj_parse_iter(fj_state *st, int implicit_root) {
       fj_skip_ws_comments(st);
       b = fj_byte(st);
       if (b == ',') { /* collapsing separator: skip empty member */
-        if (st->collect_warnings && !vss) fj_warn(st, fj_sym_empty_slot, "extra comma, collapsed an empty slot");
+        if (st->on_warning != Qnil && !vss) fj_warn(st, fj_sym_empty_slot, "extra comma, collapsed an empty slot");
         vss = 0;
         fj_advance(st, 1);
         continue;
@@ -1323,7 +1325,7 @@ static VALUE fj_parse_iter(fj_state *st, int implicit_root) {
       fj_skip_ws_comments(st);
       b = fj_byte(st);
       if (b == ',') { /* collapsing separator: skip empty slot */
-        if (st->collect_warnings && !vss) fj_warn(st, fj_sym_empty_slot, "extra comma, collapsed an empty slot");
+        if (st->on_warning != Qnil && !vss) fj_warn(st, fj_sym_empty_slot, "extra comma, collapsed an empty slot");
         vss = 0;
         fj_advance(st, 1);
         continue;
@@ -1412,8 +1414,7 @@ static VALUE fj_parse_c(VALUE self, VALUE input, VALUE opts) {
     else st.bigdecimal_load = 1; /* :auto (default), including nil */
   }
 
-  st.collect_warnings = RTEST(rb_hash_aref(opts, ID2SYM(rb_intern("warnings"))));
-  st.warnings = st.collect_warnings ? rb_ary_new() : Qnil;
+  st.on_warning = rb_hash_aref(opts, ID2SYM(rb_intern("on_warning"))); /* Qnil when absent */
 
   if (st.len >= 3 && (unsigned char)st.buf[0] == 0xEF &&
       (unsigned char)st.buf[1] == 0xBB && (unsigned char)st.buf[2] == 0xBF) {
@@ -1439,10 +1440,10 @@ static VALUE fj_parse_c(VALUE self, VALUE input, VALUE opts) {
    * whitespace / newline / concatenation do), so a bracketless comma list still
    * raises in fj_parse_iter — the unsupported implicit-root array. */
   fj_skip_ws_comments(&st);
-  if (fj_eof(&st)) return st.collect_warnings ? rb_assoc_new(Qnil, st.warnings) : Qnil;
+  if (fj_eof(&st)) return Qnil;
   value = fj_parse_iter(&st, fj_implicit_root_ahead(&st));
   fj_skip_ws_comments(&st);
-  if (fj_eof(&st)) return st.collect_warnings ? rb_assoc_new(value, st.warnings) : value;
+  if (fj_eof(&st)) return value;
   {
     VALUE arr = rb_ary_new();
     rb_ary_push(arr, value);
@@ -1450,7 +1451,7 @@ static VALUE fj_parse_c(VALUE self, VALUE input, VALUE opts) {
       rb_ary_push(arr, fj_parse_iter(&st, fj_implicit_root_ahead(&st)));
       fj_skip_ws_comments(&st);
     } while (!fj_eof(&st));
-    return st.collect_warnings ? rb_assoc_new(arr, st.warnings) : arr;
+    return arr;
   }
 }
 
@@ -1463,6 +1464,7 @@ void Init_smarter_json(void) {
   fj_to_sym_id = rb_intern("to_sym");
   fj_key_p_id = rb_intern("key?");
   fj_new_id = rb_intern("new");
+  fj_call_id = rb_intern("call");
   fj_sym_empty_slot = ID2SYM(rb_intern("empty_slot"));
   fj_sym_empty_value = ID2SYM(rb_intern("empty_value"));
   fj_sym_duplicate_key = ID2SYM(rb_intern("duplicate_key"));

data/lib/smarter_json/parser.rb

@@ -22,9 +22,9 @@ module SmarterJSON
   # stream as newline-delimited documents (NDJSON / JSONL), one per line.
   def process(input, options = {}, &block)
     if input.is_a?(String)
-      process_content(input, options, &block)
+      Recovery.process_string(input, options, &block)
     elsif input.respond_to?(:read)
-      block ? stream_io(input, options, &block) : process_content(input.read, options)
+      block ? stream_io(input, options, &block) : process(input.read, options)
     else
       raise ArgumentError, "SmarterJSON.process expects a String or an IO, got #{input.class}"
     end
@@ -43,7 +43,7 @@ module SmarterJSON
     if block
       File.open(path, "r:#{encoding}") { |io| stream_io(io, options, &block) }
     else
-      process_content(File.read(path, encoding: encoding), options)
+      process(File.read(path, encoding: encoding), options)
     end
   end
 
@@ -57,10 +57,9 @@ module SmarterJSON
         Parser.new(input, options).each_value(&block)
       end
     elsif options.fetch(:acceleration, true) && HAS_ACCELERATION
-      parse_c(input, options) # returns [result, warnings] when options[:warnings]
+      parse_c(input, options)
     else
-      parser = Parser.new(input, options)
-      options.fetch(:warnings, false) ? [parser.parse, parser.warnings] : parser.parse
+      Parser.new(input, options).parse
     end
   end
 
@@ -68,12 +67,230 @@ module SmarterJSON
   # each — bounded memory. Newline-delimited (NDJSON / JSONL); a single document
   # spanning multiple lines is not supported by the streaming path.
   def stream_io(io, options, &block)
-    io.each_line("\n") { |line| process_content(line, options, &block) }
-    nil
+    Recovery.process_string(io.read, options, &block)
   end
 
   private_class_method :process_content, :stream_io
 
+  module Recovery
+    module_function
+
+    def process_string(input, options, &block)
+      return SmarterJSON.send(:process_content, input, options, &block) unless input.valid_encoding?
+
+      if wrapper_hint?(input)
+        payloads = extract_payloads(input, options)
+        return replay_payloads(payloads, options, &block) unless payloads.empty?
+      end
+
+      SmarterJSON.send(:process_content, input, options, &block)
+    rescue ParseError => e
+      raise if e.is_a?(EncodingError)
+
+      payloads = extract_payloads(input, options)
+      return replay_payloads(payloads, options, &block) unless payloads.empty?
+
+      raise
+    end
+
+    def wrapper_hint?(input)
+      return false unless input.valid_encoding?
+
+      input.match?(/```|<json\b|BEGIN_JSON\b/i) || input.match?(/\A[[:space:]]*(?:JSON|Final answer)[[:space:]]*:/i)
+    end
+
+    def replay_payloads(payloads, options, &block)
+      handler = options[:on_warning]
+      emit_wrapper_warnings(payloads, handler)
+
+      results = payloads.map do |payload|
+        SmarterJSON.send(:process_content, payload[:slice], options)
+      end
+
+      return results.each(&block).then { nil } if block_given?
+      return nil if results.empty?
+      return results.first if results.length == 1
+
+      results
+    end
+
+    def emit_wrapper_warnings(payloads, handler)
+      return unless handler
+
+      meta = payloads.first[:meta]
+      warn(handler, :prefix_text_ignored, "ignored non-JSON text before the payload", *meta[:first_pos]) if meta[:prefix]
+      warn(handler, :code_fence_stripped, "stripped markdown code fences around the payload", *meta[:first_pos]) if meta[:fence]
+      warn(handler, :wrapper_tag_stripped, "stripped wrapper tags around the payload", *meta[:first_pos]) if meta[:wrapper]
+      warn(handler, :suffix_text_ignored, "ignored non-JSON text after the payload", *meta[:last_pos]) if meta[:suffix]
+    end
+
+    def extract_payloads(input, options)
+      payloads = candidate_ranges(input).filter_map do |range|
+        slice = input.byteslice(range.begin, range.end - range.begin)
+        begin
+          SmarterJSON.send(:process_content, slice, options.merge(on_warning: nil))
+          { slice: slice, range: range }
+        rescue ParseError
+          nil
+        end
+      end
+      meta = wrapper_meta(input, payloads.map { |p| p[:range] })
+      payloads.each { |payload| payload[:meta] = meta }
+      payloads
+    end
+
+    def wrapper_meta(input, ranges)
+      return { prefix: false, suffix: false, fence: false, wrapper: false } if ranges.empty?
+
+      first = ranges.first
+      last = ranges.last
+      prefix = input.byteslice(0, first.begin)
+      suffix = input.byteslice(last.end, input.bytesize - last.end)
+      {
+        prefix: substantive_text?(prefix),
+        suffix: substantive_text?(suffix),
+        fence: input.match?(/```/),
+        wrapper: input.match?(/<json\b|BEGIN_JSON\b/i),
+        first_pos: line_col_for(input, first.begin),
+        last_pos: line_col_for(input, last.begin)
+      }
+    end
+
+    def line_col_for(input, offset)
+      line = 1
+      col = 1
+      i = 0
+      while i < offset
+        b = input.getbyte(i)
+        break if b.nil?
+
+        if b == 0x0A
+          line += 1
+          col = 1
+          i += 1
+        elsif b == 0x0D
+          line += 1
+          col = 1
+          i += 1
+          i += 1 if i < offset && input.getbyte(i) == 0x0A
+        else
+          col += 1
+          i += 1
+        end
+      end
+      [line, col]
+    end
+
+    def substantive_text?(text)
+      return false if text.nil? || text.empty?
+
+      stripped = text.dup
+      stripped.gsub!(%r{/\*.*?\*/}m, "")
+      stripped.gsub!(/^\s*(?:#|\/\/).*$/, "")
+      !stripped.strip.empty? && !stripped.strip.match?(/\A(?:```[a-zA-Z0-9_-]*)?\z/) && !stripped.strip.match?(/\A(?:<\/?json>|BEGIN_JSON|END_JSON)\z/i)
+    end
+
+    def warn(handler, type, message, line, col)
+      handler.call(Warning.new(type, message, line, col))
+    end
+
+    def candidate_ranges(input)
+      ranges = []
+      stack = []
+      start_pos = nil
+      i = 0
+      mode = nil
+      while i < input.bytesize
+        b = input.getbyte(i)
+        if mode == :double
+          if b == 0x5C
+            i += 2
+            next
+          elsif b == 0x22
+            mode = nil
+          end
+          i += 1
+          next
+        elsif mode == :single
+          if b == 0x5C
+            i += 2
+            next
+          elsif b == 0x27
+            mode = nil
+          end
+          i += 1
+          next
+        elsif mode == :triple
+          if input.byteslice(i, 3) == "'''"
+            mode = nil
+            i += 3
+          else
+            i += 1
+          end
+          next
+        elsif mode == :line_comment
+          if [0x0A, 0x0D].include?(b)
+            mode = nil
+          else
+            i += 1
+            next
+          end
+        elsif mode == :block_comment
+          if input.byteslice(i, 2) == "*/"
+            mode = nil
+            i += 2
+          else
+            i += 1
+          end
+          next
+        else
+          if input.byteslice(i, 2) == "//"
+            mode = :line_comment
+            i += 2
+            next
+          elsif input.byteslice(i, 2) == "/*"
+            mode = :block_comment
+            i += 2
+            next
+          elsif b == 0x23
+            mode = :line_comment
+            i += 1
+            next
+          elsif b == 0x22
+            mode = :double
+            i += 1
+            next
+          elsif input.byteslice(i, 3) == "'''"
+            mode = :triple
+            i += 3
+            next
+          elsif b == 0x27
+            mode = :single
+            i += 1
+            next
+          elsif [0x7B, 0x5B].include?(b)
+            start_pos = i if stack.empty?
+            stack << b
+          elsif b == 0x7D
+            stack.pop if stack.last == 0x7B
+            if stack.empty? && start_pos
+              ranges << (start_pos...(i + 1))
+              start_pos = nil
+            end
+          elsif b == 0x5D
+            stack.pop if stack.last == 0x5B
+            if stack.empty? && start_pos
+              ranges << (start_pos...(i + 1))
+              start_pos = nil
+            end
+          end
+        end
+        i += 1
+      end
+      ranges
+    end
+  end
+
   # Hand-rolled FSM single-pass parser.
   # Layer 1: strict JSON (RFC 8259).
   # Layer 2: JSON5 additions — line/block comments, trailing comma,
@@ -143,14 +360,9 @@ module SmarterJSON
       symbolize_keys: false, # Symbol keys instead of String
       duplicate_key: :last_wins, # :last_wins | :first_wins | :raise
       bigdecimal_load: :auto, # :auto | :float | :bigdecimal (Oj-compatible)
-      warnings: false, # collect non-fatal lenient fixes; process returns [result, warnings]
+      on_warning: nil, # a callable invoked once per non-fatal lenient fix (a SmarterJSON::Warning)
     }.freeze
 
-    # Warnings collected during the parse (empty slots, empty values, dropped duplicate
-    # keys). Empty unless the parser was built with warnings: true. Public so the module
-    # functions can read it after parse / each_value.
-    attr_reader :warnings
-
     def initialize(input, options = {})
       raise ArgumentError, "input must be a String" unless input.is_a?(String)
 
@@ -158,8 +370,7 @@ module SmarterJSON
       @symbolize_keys  = opts[:symbolize_keys]
       @duplicate_key   = opts[:duplicate_key]
       @bigdecimal_load = opts[:bigdecimal_load]
-      @collect_warnings = opts[:warnings]
-      @warnings = []
+      @on_warning      = opts[:on_warning]
 
       encoding = opts[:encoding]
       @input = encoding ? input.dup.force_encoding(encoding) : input
@@ -263,7 +474,7 @@ module SmarterJSON
           # Commas are collapsing separators inside a container: an empty slot (leading,
           # interior, or trailing comma) adds nothing. Skip it; the next iteration reads
           # the following value/key or the closing bracket.
-          warn(:empty_slot, "extra comma — collapsed an empty slot") unless vss
+          warn(:empty_slot, "extra comma — collapsed an empty slot") if @on_warning && !vss
           vss = false
           advance(1)
         elsif cur_obj
@@ -300,7 +511,7 @@ module SmarterJSON
             elsif [RBRACE, COMMA].include?(b)
               # key with a colon but no value -> null (don't consume } or ,; the loop does)
               store_member(cur, key, nil)
-              warn(:empty_value, "key #{key.inspect} had no value — used null")
+              warn(:empty_value, "key #{key.inspect} had no value — used null") if @on_warning
               vss = true
             elsif b.nil?
               raise error("unexpected end of input")
@@ -573,7 +784,7 @@ module SmarterJSON
       if hash.key?(k)
         raise error("duplicate key #{k.inspect}") if @duplicate_key == :raise
 
-        warn(:duplicate_key, "duplicate key #{k.inspect} — #{@duplicate_key}")
+        warn(:duplicate_key, "duplicate key #{k.inspect} — #{@duplicate_key}") if @on_warning
         return if @duplicate_key == :first_wins
       end
       hash[k] = value
@@ -933,11 +1144,14 @@ module SmarterJSON
       value
     end
 
-    # Record a non-fatal lenient fix (only when built with warnings: true).
+    # Report a non-fatal lenient fix to the on_warning callable. The call-site guards
+    # (`if @on_warning`) keep the message string from being built on the fast path; this
+    # internal guard is the safety net so a forgotten call-site guard can't crash a
+    # handler-less caller.
     def warn(type, message)
-      return unless @collect_warnings
+      return unless @on_warning
 
-      @warnings << Warning.new(type, message, @line, @col)
+      @on_warning.call(Warning.new(type, message, @line, @col))
     end
 
     def error(message)

data/lib/smarter_json/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SmarterJSON
-  VERSION = "0.6.0"
+  VERSION = "0.8.0"
 end

data/lib/smarter_json/warning.rb

@@ -3,8 +3,8 @@
 module SmarterJSON
   # A non-fatal thing the parser worked around while staying lenient — e.g. an empty
   # comma slot it collapsed, a key with no value it read as null, or a duplicate key
-  # it dropped. Surfaced only when process / process_file is called with warnings: true
-  # (which then returns [result, warnings]); otherwise the parser stays silent.
+  # it dropped. Passed to the on_warning: callable (when process / process_file is given
+  # one) once per fix; otherwise the parser stays silent and builds no Warning at all.
   #
   #   type    — a Symbol you can branch on (:empty_slot, :empty_value, :duplicate_key)
   #   message — human-readable description

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: smarter_json
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.8.0
 platform: ruby
 authors:
 - Tilo Sloboda
 bindir: exe
 cert_chain: []
-date: 2026-06-02 00:00:00.000000000 Z
+date: 2026-06-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bigdecimal