smarter_json 1.0.0 → 1.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 84a73a6cf0785c67eb2dfaf87dc663b860d5afed6bf0816f861b6430d1f55475
-  data.tar.gz: 953aebf65ab855450a7d3b41c826c169242dd19a190f089da3f52df2da0b0a44
+  metadata.gz: 480227c64ed99ba271fe95c6e7c79be6871f6bbf5c7f8f03d9bf118bb6bd7051
+  data.tar.gz: 966190c8f2e316e3664e381bf738831259b0e469a783f2fa3a25edf5f198d41b
 SHA512:
-  metadata.gz: 8fe6e07fd99f1557a716fc37370dfc3c5cdb34e054d07fa812983d6cefea1e74108bb0e708fdb42f47a31ff8435aa0d6b8c80deeebab09a221bbd9992691be28
-  data.tar.gz: ec166df8863b136abc38df844bba2286b542de03f24d410b6e4a5914bfcf2165337add25c5731b92540328c5496ad4ec658243e21e4830687de9f44dcaaf4d54
+  metadata.gz: d53d1a84aa5ce83a2243a9adcdcddc284933dc8c357e08c8aaaec8e5e005164dfdec37d70c41bce0b2a3abf22dc7902e062adbbe3ffaa947b2232836bd01a86c
+  data.tar.gz: ae44f83437fe6c75227822a234658abb2157caf0378c8b998bab25aa40b4f1286da9a85a482be04b1be4ab17b142779140d0892ada81ae7d4342a0b7f706e9b7

data/CHANGELOG.md

@@ -1,22 +1,32 @@
 
 # SmarterJSON Change Log
 
-> ⚠️ **New Interface (since 0.9.7):**
->
-> SmarterJSON **always returns an `Array`** of documents.
+> ⚠️ SmarterJSON **always returns an `Array`** of documents.
 > 
-> `SmarterJSON.process` / `SmarterJSON.process_file` return:
->
-> — `[]` for no doc
-> - `[doc]` for one doc
-> - `[d1, d2, …]` for several docs (NDJSON / JSONL / concatenated docs)
+> `SmarterJSON.process` / `SmarterJSON.process_file`
+> both return:
+>   — `[]` for no doc
+>   - `[doc]` for one doc
+>   - `[d1, d2, …]` for several docs (NDJSON / JSONL / concatenated docs)
 
 > ⚠️ We discourage the use of `process(input).first` / `process(input)[0]` because it silently drops potential additional documents
 >    Please use `process_one` if you are expecting only one JSON doc, e.g. in API payloads.
 
+## 1.1.1 (2026-06-11)
+
+RSpec tests: 1,070 → 1,097
+
+- The C extension now emits the same `on_warning` warnings as the pure-Ruby parser. `empty_value` and `duplicate_key` warnings name the offending key (and `duplicate_key` names the resolution strategy), and the warning text, line, and column are now identical whether or not the C extension is loaded.
+
+## 1.1.0 (2026-06-09)
+
+RSpec tests: 1,038 → 1,070
+
+- New `SmarterJSON.foreach(source)` — the streaming, composable sibling of `process_file`. `source` is a file path or an IO (a socket, `StringIO`, open `File`). Without a block it returns a plain `Enumerator` (like `CSV.foreach`) that reads one document at a time, never loading the whole file, so a large NDJSON / JSONL stream can be filtered or transformed with `.select` / `.map` / `.lazy` / `.first`; with a block it streams and returns the document count, like `process_file`.
+
 ## 1.0.0 (2026-06-08)
 
-RSpec tests: 1,034
+RSpec tests: 1,038
 
 - **The public interface is now stable** — `process`, `process_one`, `process_file`, `generate`, and the documented options; semantic versioning from here on.
 - Unknown or wrongly-typed options now raise `ArgumentError` instead of being silently ignored, so a typo (e.g. `symbolize_names:` instead of `symbolize_keys:`) is caught immediately.

data/README.md

@@ -2,12 +2,25 @@
 
 ![Gem Version](https://img.shields.io/gem/v/smarter_json) [![codecov](https://codecov.io/gh/tilo/smarter_json/branch/main/graph/badge.svg)](https://codecov.io/gh/tilo/smarter_json) <!-- [![Downloads](https://img.shields.io/gem/dt/smarter_json)](https://rubygems.org/gems/smarter_json) --> [![RubyGems](https://img.shields.io/badge/RubyGems-smarter__json-brightgreen?logo=rubygems&logoColor=white)](https://rubygems.org/gems/smarter_json) [![Ruby Toolbox](https://img.shields.io/badge/Ruby%20Toolbox-smarter__json-brightgreen)](https://www.ruby-toolbox.com/projects/smarter_json)
 
-A lenient, fast JSON processor for Ruby. It extracts strict JSON, NDJSON, JSON5, HJSON-style config, and the messy JSON-ish input humans actually write — and in benchmarks it matches or beats Oj on every file. SmarterJSON is opinionated: we want your JSON processing to be successful. Traditional JSON parsers are strict - they stop at the first deviation - SmarterJSON keeps going - it optimizes for getting your data out, not for policing the JSON spec.
+A lenient, fast JSON processor for Ruby. It extracts strict JSON, NDJSON, JSONL, JSON5, HJSON-style config, and the messy JSON-ish input humans actually write — and in benchmarks it matches or beats Oj on every file. SmarterJSON is opinionated: we want your JSON processing to be successful. Traditional JSON parsers are strict - they stop at the first deviation - SmarterJSON keeps going - it optimizes for getting your data out, not for policing the JSON spec.
 
 > **SmarterJSON: one tool, no modes — want strict? Please use the stdlib `json` gem.**
 
+## Features at a glance
+
+- **Reads the whole human-JSON superset, no modes or flags** — strict JSON, NDJSON, JSONL, JSON5, HJSON, JSONC, plus comments, trailing commas, unquoted / single / triple / smart quotes, an implicit root object, `NaN` / `Infinity` / hex / underscores, Python & JavaScript literals, a UTF-8 BOM, mixed line endings, and any Ruby encoding (see [What it accepts](#what-it-accepts-beyond-strict-json) for the full list).
+- **Every document from multi-document input, in one call** — `process` returns an `Array` of all of them; `process_one` returns the single value and warns if there was more than one (never raises; routed to `on_warning`, else `Rails.logger`, else `Kernel.warn`).
+- **Streaming in bounded memory** — pass a block, or use `foreach(path_or_io)` for a composable `Enumerator` you can `.select` / `.map` / `.lazy` over.
+- **Recovers JSON from LLM / markdown noise** — strips markdown code fences, surrounding prose, and `<json>` tags, and pulls every payload out of one messy blob.
+- **Writes JSON too** — `generate` with pretty-printing, NDJSON, `sort_keys`, `ascii_only`, `script_safe`, `allow_nan`, and `coerce` (via `as_json`); iterative, so deeply nested data is depth-safe.
+- **Keeps number precision** — `BigDecimal` by default (Oj-compatible), or `:float` / `:auto`.
+- **Transparent leniency** — pass an optional `on_warning` callback to be handed every lenient fix (an empty slot collapsed, a duplicate key dropped, a code fence stripped, …); with no handler the parser stays silent and adds zero overhead.
+- **Fast, and runs everywhere** — a C extension that matches or beats Oj, with a pure-Ruby fallback for platforms that can't build it. Stable, semantically versioned, thread-safe, Ruby 2.6+.
+
 ## Why SmarterJSON?
 
+> 📖 **The thinking behind it:** [*Strict by Accident: Your JSON parser isn't broken, it's answering the wrong question*](https://dev.to/tilo_sloboda/strict-by-accident-your-json-parser-isnt-broken-its-answering-the-wrong-question-54f0) — why a data pipeline wants a lenient, recovery-first parser rather than a spec-policing one.
+
 **Are you tired of seeing errors like these?**
 
 ```
@@ -40,7 +53,7 @@ A lenient, fast JSON processor for Ruby. It extracts strict JSON, NDJSON, JSON5,
 Traditional JSON parsers reject anything that isn't perfectly strict JSON. That means your code breaks on malformed data.
 
 SmarterJSON is built on the opposite principle: **you shouldn't have to care what flavor of JSON you were handed** and **you shouldn't lose the whole document because of formatting errors.**
-Give it strict JSON, NDJSON, JSON5, an HJSON-style config file, LLM-generated JSON, or a copy-pasted blob with comments and trailing commas — it just extracts the data from it.
+Give it strict JSON, NDJSON, JSONL, JSON5, an HJSON-style config file, LLM-generated JSON, or a copy-pasted blob with comments and trailing commas — it just extracts the data from it.
 When it is lenient, `smarter_json` isn't dropping data that exists — it's just not raising an eyebrow at a suspicious gap (like an extra comma).
 
 A strict parser would refuse the whole document and recover nothing; `smarter_json` returns everything except the formatting error.
@@ -73,13 +86,15 @@ It raises only on genuinely unreadable input (unterminated string, mismatched br
 The lenient grammar is a superset of these human-JSON specs — listed once, here:
 
 * [JSON5](https://json5.org/)
-* [HJSON](https://hjson.github.io/)
+* [HJSON](https://hjson.github.io/) <sup>†</sup>
 * [JWCC / HuJSON](https://github.com/tailscale/hujson)
 * [Nigel Tao](https://nigeltao.github.io/blog/2021/json-with-commas-comments.html)
 * [JSONH](https://github.com/jsonh-org/Jsonh)
 * [JSONC (VS Code)](https://jsonc.org/)
 * [NDJSON / JSON Text Sequences (RFC 7464)](https://datatracker.ietf.org/doc/html/rfc7464).
 
+<sup>†</sup> A deliberate subset. SmarterJSON's quoteless (unquoted) string values are single-line — it does **not** parse HJSON's unquoted multi-line strings; use a quoted or triple-quoted (`'''…'''`) string for multiline. This is by design: SmarterJSON is one deterministic, no-modes superset of the JSON-family dialects (JSON5 / HJSON / JSONC / …), so it adopts a feature only where it does not conflict with the others — and an unquoted string that may span newlines collides with newline-as-a-document-separator (NDJSON, implicit-root config), so it is left out.
+
 ## Installation
 
 ```ruby
@@ -130,7 +145,7 @@ See [Examples](#examples) below for multi-document input, streaming, and recover
 
 ## Stable interface & thread safety
 
-The public interface is now considered stable: `SmarterJSON.process`, `SmarterJSON.process_one`, `SmarterJSON.process_file`, `SmarterJSON.generate`, and the documented options in this README/docs are the supported surface.
+The public interface is: `SmarterJSON.process`, `SmarterJSON.process_one`, `SmarterJSON.process_file`, `SmarterJSON.foreach`, `SmarterJSON.generate`, and the documented options in this README/docs are the supported surface. `SmarterJSON.process` and `SmarterJSON.process_file` always return an `Array` of documents; `process_one` returns the single document's value (or `nil`), and emits a warning if there is more than one doc.
 
 Concurrent calls are safe. The processor and generator keep per-call state local, and the C extension only caches Ruby IDs / constants at load time; it does not share mutable state across calls.
 
@@ -243,6 +258,46 @@ For input larger than memory, pass a block: each document is yielded as it is re
 SmarterJSON.process_file("events.ndjson") { |event| EventJob.perform_async(event) }
 ```
 
+**Try it on a file you already have.** SmarterJSON reads **NDJSON / JSONL natively** — and Claude Code stores every session as a JSONL transcript (`~/.claude/projects/<project>/<session-id>.jsonl`, one JSON document per line). Walk yours, one record at a time:
+
+```ruby
+require "awesome_print" # optional — readable nested output
+
+SmarterJSON.process_file("#{Dir.home}/.claude/projects/<project>/<session-id>.jsonl") do |entry|
+  ap entry              # each line is a full document — a message, a tool call, a result, …
+  puts "-" * 80
+end
+```
+
+### Filtering and rewriting a large file (`foreach`)
+
+`SmarterJSON.foreach(source)` is the composable sibling of `process_file`. `source` is a file path or any IO (a socket, a `StringIO`, an open `File`). With no block it returns a plain `Enumerator` (like `CSV.foreach`) that reads one document at a time, so you can chain `.select` / `.map` and friends. Add `.lazy` to keep the whole chain bounded in memory, even when the filtered set is large:
+
+```ruby
+# Keep only the user/assistant turns of a transcript — one document in memory at a time
+SmarterJSON.foreach("session.jsonl", symbolize_keys: true)
+           .lazy
+           .select { |doc| %w[user assistant].include?(doc[:type]) }
+           .each   { |doc| puts doc[:text] }
+```
+
+Because it streams both ends, you can **filter a big file down and rewrite it** without ever loading the whole thing:
+
+```ruby
+File.open("filtered.jsonl", "w") do |out|
+  SmarterJSON.foreach("session.jsonl", symbolize_keys: true)
+             .lazy
+             .select { |doc| %w[user assistant].include?(doc[:type]) }
+             .each   { |doc| out.puts SmarterJSON.generate(doc) }
+end
+```
+
+Pass an IO instead of a path to stream straight from a socket or an HTTP response body — anything `IO`-like works (an IO is single-pass, read once):
+
+```ruby
+SmarterJSON.foreach(response_io).each { |event| handle(event) }
+```
+
 ### Recovering JSON from LLM / markdown noise
 
 When the payload is wrapped in markdown fences, surrounding prose, or tags, `process` (or `process_one` for a single payload) strips the wrapper and reads what's inside. (Clean JSON never pays for this — recovery only runs when a straight read fails.)

data/docs/basic_read_api.md

@@ -103,6 +103,28 @@ SmarterJSON.process(io) { |doc| handle(doc) }
 
 The streaming path now frames whole top-level documents, not just one line at a time. That means NDJSON / JSONL still work, but pretty-printed multi-line objects and arrays work too, as do mixed `\n` / `\r\n` / `\r` line endings and comment-only separators between documents.
 
+## `SmarterJSON.foreach` — stream a file or IO, composably
+
+`foreach` is the composable sibling of `process_file`. Its argument is a **file path or any IO** (a socket, a `StringIO`, an open `File`); a String is always a path, never content.
+
+With a block it behaves exactly like the block form above — streams each document, returns the **document count**. Without a block it returns a plain `Enumerator` (like `CSV.foreach` — **not** an `Enumerator::Lazy`), so `.map` / `.select` return Arrays the usual way, and you can chain over the stream:
+
+```ruby
+SmarterJSON.foreach("events.ndjson").each { |event| EventJob.perform_async(event) }   # like the block form
+SmarterJSON.foreach("events.ndjson").select { |e| e["level"] == "error" }              # => an Array of the matches
+```
+
+It reads one document at a time, so `foreach(path).first(3)` only reads ~3 documents off disk, and `.next` pulls them one by one. `.map` / `.select` read the source lazily but still build an Array of their *result*; to keep a whole pipeline bounded end to end (a large filtered set off a fat file), add `.lazy` at the call site:
+
+```ruby
+SmarterJSON.foreach("session.jsonl", symbolize_keys: true)
+           .lazy
+           .select { |doc| %w[user assistant].include?(doc[:type]) }
+           .each   { |doc| puts doc[:text] }
+```
+
+Options are validated eagerly — a bad option key or value raises immediately, before any iteration. An **IO source is single-pass** (an IO can only be read once), so iterating the returned Enumerator a second time over the same IO yields nothing; a path-backed `foreach` re-opens the file and is re-iterable.
+
 ## The C extension and the pure-Ruby fallback
 
 By default (`acceleration: true`) the C extension is used when it is compiled and loadable (`SmarterJSON::HAS_ACCELERATION` is then `true`); otherwise the pure-Ruby implementation runs and produces identical results. Pass `acceleration: false` to force the pure-Ruby path. See [Configuration Options](./options.md).

data/docs/examples.md

@@ -83,6 +83,28 @@ For input larger than memory, pass a block. Each recovered document is yielded o
 SmarterJSON.process_file("events.ndjson") { |event| EventJob.perform_async(event) }
 ```
 
+**A JSONL file you already have:** Claude Code stores each session as a JSONL transcript — `~/.claude/projects/<project>/<session-id>.jsonl`, one JSON document per line (a message, a tool call, a result, …). It reads the same way, one record at a time:
+
+```ruby
+require "awesome_print" # optional — for readable nested output
+
+SmarterJSON.process_file("#{Dir.home}/.claude/projects/<project>/<session-id>.jsonl") do |entry|
+  ap entry              # each line is a full document
+  puts "-" * 80
+end
+```
+
+**Filter and rewrite as a stream — `SmarterJSON.foreach`:** `foreach(source)` is the composable sibling of `process_file`; `source` is a file path or any IO (a socket, a `StringIO`, an open `File`). Without a block it returns a plain `Enumerator` (like `CSV.foreach`) that reads one document at a time, so it chains with `.select` / `.map`; add `.lazy` to keep the whole pipeline bounded in memory. This filters a transcript down to its user/assistant turns and writes a smaller file, never loading all of it:
+
+```ruby
+File.open("filtered.jsonl", "w") do |out|
+  SmarterJSON.foreach("session.jsonl", symbolize_keys: true)
+             .lazy
+             .select { |doc| %w[user assistant].include?(doc[:type]) }
+             .each   { |doc| out.puts SmarterJSON.generate(doc) }
+end
+```
+
 ### Example 6: Symbolize Keys
 
 ```ruby

data/ext/smarter_json/smarter_json.c

@@ -81,9 +81,9 @@ typedef struct {
  * an error) by scanning from the start of the buffer. CR, LF, and CRLF each
  * count as one newline; col is bytes since the last line start (1-based).
  * Keeping this off the hot path is the point — fj_advance never touches it. */
-static void fj_line_col(fj_state *st, long *line, long *col) {
+static void fj_line_col_to(fj_state *st, long stop, long *line, long *col) {
   long l = 1, c = 1, i;
-  long limit = (st->pos < st->len) ? st->pos : st->len;
+  long limit = (stop < st->len) ? stop : st->len;
   for (i = 0; i < limit; i++) {
     unsigned char b = (unsigned char)st->buf[i];
     if (b == 0x0A) { l++; c = 1; }
@@ -93,6 +93,9 @@ static void fj_line_col(fj_state *st, long *line, long *col) {
   *line = l;
   *col = c;
 }
+static void fj_line_col(fj_state *st, long *line, long *col) {
+  fj_line_col_to(st, st->pos, line, col);
+}
 
 /* Report a non-fatal lenient fix to the on_warning callable — a no-op (and builds no
  * Warning) when no handler was given. The internal Qnil guard is the safety net; the
@@ -106,6 +109,25 @@ static void fj_warn(fj_state *st, VALUE type_sym, const char *msg) {
                         rb_utf8_str_new_cstr(msg), LONG2NUM(line), LONG2NUM(col)));
 }
 
+/* Like fj_warn but the message is a prebuilt Ruby String (rb_enc_sprintf, for messages
+ * that interpolate the offending key). Same Qnil guard and st->pos line/col. */
+static void fj_warn_str(fj_state *st, VALUE type_sym, VALUE msg) {
+  long line, col;
+  if (st->on_warning == Qnil) return;
+  fj_line_col(st, &line, &col);
+  rb_funcall(st->on_warning, fj_call_id, 1,
+             rb_funcall(cWarning, fj_new_id, 4, type_sym, msg, LONG2NUM(line), LONG2NUM(col)));
+}
+
+/* Like fj_warn_str but at an explicit (line,col) — for a warning detected away from the
+ * cursor. Duplicate keys are found while building the closed object, but attributed to
+ * the object's closing position so the column matches the pure-Ruby parser. */
+static void fj_warn_str_at(fj_state *st, VALUE type_sym, VALUE msg, long line, long col) {
+  if (st->on_warning == Qnil) return;
+  rb_funcall(st->on_warning, fj_call_id, 1,
+             rb_funcall(cWarning, fj_new_id, 4, type_sym, msg, LONG2NUM(line), LONG2NUM(col)));
+}
+
 /* 1-based column of the current byte position (bytes since the last line start).
  * Used for triple-quoted indentation stripping (smarter_json.md §2.3). */
 static long fj_column(fj_state *st) {
@@ -1290,7 +1312,7 @@ void rb_hash_bulk_insert(long, const VALUE *, VALUE);
 /* Build a Hash from `count` interleaved key,value slots. Fast path (String keys,
  * default :last_wins): pre-size + bulk insert. symbolize_keys / :first_wins use a
  * per-member loop into the same pre-sized hash. */
-static VALUE fj_build_object(fj_state *st, const VALUE *pairs, long count) {
+static VALUE fj_build_object(fj_state *st, const VALUE *pairs, const long *positions, long count) {
   long  entries = count / 2, i;
   VALUE hash    = rb_hash_new_capa(entries);
 
@@ -1305,7 +1327,16 @@ static VALUE fj_build_object(fj_state *st, const VALUE *pairs, long count) {
     VALUE k = st->symbolize_keys ? rb_funcall(pairs[i], fj_to_sym_id, 0) : pairs[i];
     if (st->dup_first_wins || st->on_warning != Qnil) {
       if (RTEST(rb_funcall(hash, fj_key_p_id, 1, k))) {
-        fj_warn(st, fj_sym_duplicate_key, "duplicate key");
+        if (st->on_warning != Qnil) {
+          long wl, wc;
+          fj_line_col_to(st, positions[i + 1], &wl, &wc); /* the duplicate member's value position — matches the Ruby parser */
+          fj_warn_str_at(st, fj_sym_duplicate_key,
+                         rb_enc_sprintf(rb_utf8_encoding(),
+                                        "duplicate key %"PRIsVALUE" \xe2\x80\x94 %s",
+                                        rb_inspect(k),
+                                        st->dup_first_wins ? "first_wins" : "last_wins"),
+                         wl, wc);
+        }
         if (st->dup_first_wins) continue;
       }
     }
@@ -1323,6 +1354,7 @@ typedef struct { long mark; int is_obj; } fj_frame;
 
 typedef struct {
   VALUE    *vptr;  long vhead;  long vcapa;  /* pending values (GC-marked) */
+  long     *pptr;                            /* byte offset just past each pushed value (mirrors vptr/vcapa); used only to attribute a duplicate-key warning to the duplicate member's position */
   fj_frame *fptr;  long fhead;  long fcapa;  /* open-container frames (no VALUEs) */
 } fj_pstack;
 
@@ -1334,12 +1366,15 @@ static void fj_pstack_mark(void *p) {
 static void fj_pstack_free(void *p) {
   fj_pstack *ps = (fj_pstack *)p;
   if (ps->vptr != NULL) xfree(ps->vptr);
+  if (ps->pptr != NULL) xfree(ps->pptr);
   if (ps->fptr != NULL) xfree(ps->fptr);
   xfree(ps);
 }
 static size_t fj_pstack_memsize(const void *p) {
   const fj_pstack *ps = (const fj_pstack *)p;
-  return sizeof(fj_pstack) + (size_t)ps->vcapa * sizeof(VALUE) + (size_t)ps->fcapa * sizeof(fj_frame);
+  return sizeof(fj_pstack) + (size_t)ps->vcapa * sizeof(VALUE)
+         + (ps->pptr ? (size_t)ps->vcapa * sizeof(long) : 0)
+         + (size_t)ps->fcapa * sizeof(fj_frame);
 }
 static const rb_data_type_t fj_pstack_type = {
   "smarter_json/pstack",
@@ -1347,8 +1382,15 @@ static const rb_data_type_t fj_pstack_type = {
   0, 0, RUBY_TYPED_FREE_IMMEDIATELY,
 };
 
-static inline void fj_vpush(fj_pstack *ps, VALUE v) {
-  if (ps->vhead >= ps->vcapa) { ps->vcapa *= 2; REALLOC_N(ps->vptr, VALUE, ps->vcapa); }
+static inline void fj_vpush(fj_state *st, fj_pstack *ps, VALUE v) {
+  if (ps->vhead >= ps->vcapa) {
+    ps->vcapa *= 2;
+    REALLOC_N(ps->vptr, VALUE, ps->vcapa);
+    if (ps->pptr) REALLOC_N(ps->pptr, long, ps->vcapa);
+  }
+  /* pptr is allocated only when on_warning is set; the fast path (no handler) does no
+   * extra store. The offset is just past this value — the duplicate-key warning position. */
+  if (ps->pptr) ps->pptr[ps->vhead] = st->pos;
   ps->vptr[ps->vhead++] = v;
 }
 static inline void fj_fpush(fj_pstack *ps, long mark, int is_obj) {
@@ -1368,6 +1410,7 @@ static VALUE fj_parse_iter(fj_state *st, int implicit_root) {
   int        vss = 0; /* warnings: has a value landed in the current container since the last separator? */
 
   ps->vptr = ALLOC_N(VALUE, 64);    ps->vhead = 0; ps->vcapa = 64;
+  ps->pptr = (st->on_warning != Qnil) ? ALLOC_N(long, 64) : NULL; /* only the warning path needs per-value positions */
   ps->fptr = ALLOC_N(fj_frame, 16); ps->fhead = 0; ps->fcapa = 16;
 
   if (implicit_root) fj_fpush(ps, 0, 1);
@@ -1398,7 +1441,7 @@ static VALUE fj_parse_iter(fj_state *st, int implicit_root) {
       b = fj_byte(st);
       if (FJ_UNLIKELY(fj_needs_ws_skip(b))) { fj_skip_ws_comments(st); b = fj_byte(st); }
       if (b == ',') { /* collapsing separator: skip empty member */
-        if (st->on_warning != Qnil && !vss) fj_warn(st, fj_sym_empty_slot, "extra comma, collapsed an empty slot");
+        if (st->on_warning != Qnil && !vss) fj_warn(st, fj_sym_empty_slot, "extra comma \xe2\x80\x94 collapsed an empty slot");
         vss = 0;
         fj_advance(st, 1);
         continue;
@@ -1406,17 +1449,17 @@ static VALUE fj_parse_iter(fj_state *st, int implicit_root) {
       if (b == '}') {
         VALUE hash;
         fj_advance(st, 1);
-        hash = fj_build_object(st, &ps->vptr[mark], ps->vhead - mark);
+        hash = fj_build_object(st, &ps->vptr[mark], ps->pptr ? &ps->pptr[mark] : NULL, ps->vhead - mark);
         ps->vhead = mark;
         ps->fhead--;
         if (ps->fhead == 0) { result = hash; break; }
-        fj_vpush(ps, hash);
+        fj_vpush(st, ps, hash);
         vss = 1;
         continue;
       }
       if (b == -1) {
         if (implicit_root && ps->fhead == 1) {
-          result = fj_build_object(st, &ps->vptr[mark], ps->vhead - mark);
+          result = fj_build_object(st, &ps->vptr[mark], ps->pptr ? &ps->pptr[mark] : NULL, ps->vhead - mark);
           break;
         }
         fj_error(st, "unterminated object");
@@ -1430,28 +1473,32 @@ static VALUE fj_parse_iter(fj_state *st, int implicit_root) {
       b = fj_byte(st);
       if (FJ_UNLIKELY(fj_needs_ws_skip(b))) { fj_skip_ws_comments(st); b = fj_byte(st); }
       if (b == '{' || b == '[') {
-        fj_vpush(ps, key);
+        fj_vpush(st, ps, key);
         fj_advance(st, 1);
         fj_fpush(ps, ps->vhead, (b == '{'));
         vss = 0;
         continue;
       }
       if (b == '}' || b == ',') { /* key with a colon but no value -> null */
-        fj_vpush(ps, key);
-        fj_vpush(ps, Qnil);
-        fj_warn(st, fj_sym_empty_value, "empty value, used null");
+        fj_vpush(st, ps, key);
+        fj_vpush(st, ps, Qnil);
+        if (st->on_warning != Qnil)
+          fj_warn_str(st, fj_sym_empty_value,
+                      rb_enc_sprintf(rb_utf8_encoding(),
+                                     "key %"PRIsVALUE" had no value \xe2\x80\x94 used null",
+                                     rb_inspect(key)));
         vss = 1;
         continue;
       }
       if (b == -1) fj_error(st, "unexpected end of input");
-      fj_vpush(ps, key);
-      fj_vpush(ps, fj_parse_member_value(st));
+      fj_vpush(st, ps, key);
+      fj_vpush(st, ps, fj_parse_member_value(st));
       vss = 1;
     } else { /* array */
       b = fj_byte(st);
       if (FJ_UNLIKELY(fj_needs_ws_skip(b))) { fj_skip_ws_comments(st); b = fj_byte(st); }
       if (b == ',') { /* collapsing separator: skip empty slot */
-        if (st->on_warning != Qnil && !vss) fj_warn(st, fj_sym_empty_slot, "extra comma, collapsed an empty slot");
+        if (st->on_warning != Qnil && !vss) fj_warn(st, fj_sym_empty_slot, "extra comma \xe2\x80\x94 collapsed an empty slot");
         vss = 0;
         fj_advance(st, 1);
         continue;
@@ -1463,7 +1510,7 @@ static VALUE fj_parse_iter(fj_state *st, int implicit_root) {
         ps->vhead = mark;
         ps->fhead--;
         if (ps->fhead == 0) { result = ary; break; }
-        fj_vpush(ps, ary);
+        fj_vpush(st, ps, ary);
         vss = 1;
         continue;
       }
@@ -1481,10 +1528,10 @@ static VALUE fj_parse_iter(fj_state *st, int implicit_root) {
          smart-quote, literals) falls through to the full dispatch below. */
       if (b == '-' || b == '+' || b == '.' || (b >= '0' && b <= '9')) {
         VALUE num;
-        if (fj_try_member_number(st, &num)) { fj_vpush(ps, num); vss = 1; continue; }
+        if (fj_try_member_number(st, &num)) { fj_vpush(st, ps, num); vss = 1; continue; }
       }
-      if (b == '"') { fj_vpush(ps, fj_parse_string(st, '"')); vss = 1; continue; }
-      fj_vpush(ps, fj_parse_member_value(st));
+      if (b == '"') { fj_vpush(st, ps, fj_parse_string(st, '"')); vss = 1; continue; }
+      fj_vpush(st, ps, fj_parse_member_value(st));
       vss = 1;
     }
   }

data/lib/smarter_json/parser.rb

@@ -57,6 +57,41 @@ module SmarterJSON
     end
   end
 
+  # SmarterJSON.foreach(source, options = {}) — the streaming, composable sibling of
+  # process_file, mirroring the stdlib convention (CSV.foreach / File.foreach): a
+  # plain Enumerator (NOT Enumerator::Lazy), so .map / .select behave the normal way
+  # and return an Array.
+  #
+  # `source` is a file path (opened and streamed from disk, like process_file) OR an
+  # IO — a socket, a StringIO, an open File — streamed directly from its current
+  # position. A String is always a path, never content. An IO source is single-pass:
+  # it can only be read once, so iterating the returned Enumerator a second time over
+  # the same IO yields nothing.
+  #
+  # Without a block: returns an Enumerator over each top-level document, reading one
+  # document at a time via readpartial — it never slurps the whole file the way
+  # process_file(path) does. So foreach(path).first(3) reads only ~3 documents off
+  # disk, and foreach(src).each { … } / .next stream in bounded memory. .map / .select
+  # read the source one document at a time but still build an Array of their result;
+  # for a chain that stays bounded end to end (a large filtered set off a fat file)
+  # opt into .lazy at the call site: foreach(src).lazy.select { … }.each { … }.
+  #
+  # With a block: streams each document and returns the document count — identical
+  # to process_file(path) { |doc| … } (or process(io) { |doc| … } for an IO).
+  #
+  # Options are validated eagerly (before the Enumerator is returned), so a bad
+  # option key or value fails fast rather than on first iteration.
+  def foreach(source, options = {}, &block)
+    options = Options.process_options(options)
+    return enum_for(:foreach, source, options) unless block
+
+    if source.respond_to?(:read) # an IO (socket, StringIO, open File) — stream it directly
+      stream_io(source, options, &block)
+    else                         # a path — open the file and stream from disk
+      process_file(source, options, &block)
+    end
+  end
+
   # SmarterJSON.process_one(input, options = {}) — the single-document accessor.
   #
   # Returns the first document's value (or nil when the input holds no documents).

data/lib/smarter_json/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SmarterJSON
-  VERSION = "1.0.0"
+  VERSION = "1.1.1"
 end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: smarter_json
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.1.1
 platform: ruby
 authors:
 - Tilo Sloboda
 bindir: exe
 cert_chain: []
-date: 2026-06-09 00:00:00.000000000 Z
+date: 2026-06-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bigdecimal
@@ -24,7 +24,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 description: 'A lenient, fast JSON processor for Ruby. It extracts strict JSON, NDJSON,
-  JSON5, HJSON-style config, and the messy JSON-ish input humans and LLMs actually
+  JSONL, JSON5, HJSON-style config, and the messy JSON-ish input humans and LLMs actually
   write — comments, trailing commas, single / unquoted / smart quotes, Python and
   JS keywords, a UTF-8 BOM, and more all parse to the same Ruby objects, with no modes
   or flags to set. Where a traditional parser stops at the first deviation and throws
@@ -92,6 +92,6 @@ required_rubygems_version: !ruby/object:Gem::Requirement
 requirements: []
 rubygems_version: 3.6.9
 specification_version: 4
-summary: A lenient, fast JSON processor for Ruby — reads strict JSON, NDJSON, JSON5,
-  HJSON, and the messy JSON humans and LLMs actually write.
+summary: A lenient, fast JSON processor for Ruby — reads strict JSON, NDJSON, JSONL,
+  JSON5, HJSON, and the messy JSON humans and LLMs actually write.
 test_files: []