fastsimdjson 0.3.0__tar.gz → 0.5.0__tar.gz

{fastsimdjson-0.3.0/src/fastsimdjson.egg-info → fastsimdjson-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fastsimdjson
-Version: 0.3.0
+Version: 0.5.0
 Summary: Fast JSON parsing for Python, built on simdjson
 Author-email: Daniel Lemire <daniel@lemire.me>
 License-Expression: Apache-2.0
@@ -197,6 +197,40 @@ returned, otherwise `JSONDecodeError` reports `json`'s message and the
 position in the whole input. A truncated last document is an error. The
 views returned by `parse_many` remain valid after the iterator moves on.
 
+### Streams on several threads: `threads=N` and `map_many`
+
+On free-threaded Python (3.14t and newer), a stream can be parsed by several
+threads at once:
+
+```python
+docs = fastsimdjson.loads_many(data, threads=16)        # a list, in order
+views = fastsimdjson.parse_many(data, threads=16)       # a list of views
+# fn runs in the threads too, on each document; its results, in order:
+scores = fastsimdjson.map_many(score, data, threads=16)
+```
+
+With `threads`, `loads_many` and `parse_many` return a list of all the
+documents instead of an iterator. `map_many(fn, data, **kw)` returns
+`[fn(doc) for doc in loads_many(data, **kw)]` (`parse_many` with
+`lazy=True`), with `fn` called in the threads. `threads=0` (or `None`) uses
+one thread per processor; small inputs use fewer threads (about one per
+512 KB). The input is cut into chunks at newlines (before each `\x1e` for
+`json_seq`), so `threads` works with the formats `"whitespace"`, `"lines"`
+and `"json_seq"`.
+
+The results, and the exceptions, are those of one thread: when a chunk
+contains an error or a document that only `json` accepts, the documents
+before it are kept and the rest is parsed in the calling thread, as without
+threads. An exception raised by `fn` is raised again, for the first document
+where it happened; `fn` may then have been called on later documents, in
+other threads, and it must be safe to call from several threads at once.
+
+`map_many` scales best: each document is built, used and freed in the same
+thread, and only `fn`'s results come back. `loads_many` must return every
+document, and the Python objects of all of them, to the calling thread.
+With the GIL (the default build of Python), the calling thread does all the
+work: `threads` has no effect on speed.
+
 ### `release`
 
 `release()` frees the simdjson parser and the string caches kept by the
@@ -433,14 +467,35 @@ msgspec's `decode_lines` and `loads_many` are on par (`loads_many` is about 2%
 faster). `parse_many` only creates the views; reading values from them adds to
 its time.
 
+### Streams on several threads
+
+300 MB of NDJSON (146,624 records: tweets, GitHub events, citm performances,
+gsoc and update-center entries), free-threaded Python 3.14.6, collector
+disabled (see below), Intel Xeon Gold 6548N (64 cores); GB/s:
+
+| threads | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| `map_many(len, data, threads=N)` | 0.91 | 1.77 | 3.28 | 6.22 | 10.9 | 15.4 | **21.2** |
+| `parse_many(data, threads=N)` | 1.82 | 3.06 | 4.80 | 6.90 | 10.2 | 12.3 | 16.4 |
+| `loads_many(data, threads=N)` | 0.82 | 1.43 | 2.26 | 3.88 | 5.77 | 7.45 | 8.63 |
+
+`map_many` is 23 times faster on 64 threads than on one. `loads_many` builds
+every document for the calling thread and is 11 times faster. With the
+collector enabled, Python collects after the call returns: it examines the
+millions of objects just created, which takes about 0.1 s here, with or
+without threads.
+
 ## Limitations
 
 * The simdjson parser and the key and string caches are thread-local.
   `release()` frees the parser and the cached strings retained by the calling
-  thread. A parser that grows past 64 MB is freed on its own at the end of
-  that call; its caches stay. A thread that exits without `release()` leaves
-  its cached strings behind.
+  thread, and they are freed when a Python thread exits. A parser that grows
+  past 64 MB is freed on its own at the end of that call; its caches stay.
 * The module is marked free-threading compatible (`Py_MOD_GIL_NOT_USED` on
   Python 3.13 and newer), so importing it on a free-threaded build does not
-  re-enable the GIL. On a free-threaded build, `bytearray` and `memoryview`
-  inputs are copied before parsing. Subinterpreters are not supported.
+  re-enable the GIL. It is tested with many threads parsing, serializing and
+  sharing views and stream iterators on free-threaded Python 3.14 and 3.15.
+  On a free-threaded build, `bytearray` and `memoryview` inputs are copied
+  before parsing, and `dumps` and `dumpb` hold references to the items of the
+  containers they serialize, since other threads may change them.
+  Subinterpreters are not supported.

{fastsimdjson-0.3.0 → fastsimdjson-0.5.0}/README.md

@@ -166,6 +166,40 @@ returned, otherwise `JSONDecodeError` reports `json`'s message and the
 position in the whole input. A truncated last document is an error. The
 views returned by `parse_many` remain valid after the iterator moves on.
 
+### Streams on several threads: `threads=N` and `map_many`
+
+On free-threaded Python (3.14t and newer), a stream can be parsed by several
+threads at once:
+
+```python
+docs = fastsimdjson.loads_many(data, threads=16)        # a list, in order
+views = fastsimdjson.parse_many(data, threads=16)       # a list of views
+# fn runs in the threads too, on each document; its results, in order:
+scores = fastsimdjson.map_many(score, data, threads=16)
+```
+
+With `threads`, `loads_many` and `parse_many` return a list of all the
+documents instead of an iterator. `map_many(fn, data, **kw)` returns
+`[fn(doc) for doc in loads_many(data, **kw)]` (`parse_many` with
+`lazy=True`), with `fn` called in the threads. `threads=0` (or `None`) uses
+one thread per processor; small inputs use fewer threads (about one per
+512 KB). The input is cut into chunks at newlines (before each `\x1e` for
+`json_seq`), so `threads` works with the formats `"whitespace"`, `"lines"`
+and `"json_seq"`.
+
+The results, and the exceptions, are those of one thread: when a chunk
+contains an error or a document that only `json` accepts, the documents
+before it are kept and the rest is parsed in the calling thread, as without
+threads. An exception raised by `fn` is raised again, for the first document
+where it happened; `fn` may then have been called on later documents, in
+other threads, and it must be safe to call from several threads at once.
+
+`map_many` scales best: each document is built, used and freed in the same
+thread, and only `fn`'s results come back. `loads_many` must return every
+document, and the Python objects of all of them, to the calling thread.
+With the GIL (the default build of Python), the calling thread does all the
+work: `threads` has no effect on speed.
+
 ### `release`
 
 `release()` frees the simdjson parser and the string caches kept by the
@@ -402,14 +436,35 @@ msgspec's `decode_lines` and `loads_many` are on par (`loads_many` is about 2%
 faster). `parse_many` only creates the views; reading values from them adds to
 its time.
 
+### Streams on several threads
+
+300 MB of NDJSON (146,624 records: tweets, GitHub events, citm performances,
+gsoc and update-center entries), free-threaded Python 3.14.6, collector
+disabled (see below), Intel Xeon Gold 6548N (64 cores); GB/s:
+
+| threads | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| `map_many(len, data, threads=N)` | 0.91 | 1.77 | 3.28 | 6.22 | 10.9 | 15.4 | **21.2** |
+| `parse_many(data, threads=N)` | 1.82 | 3.06 | 4.80 | 6.90 | 10.2 | 12.3 | 16.4 |
+| `loads_many(data, threads=N)` | 0.82 | 1.43 | 2.26 | 3.88 | 5.77 | 7.45 | 8.63 |
+
+`map_many` is 23 times faster on 64 threads than on one. `loads_many` builds
+every document for the calling thread and is 11 times faster. With the
+collector enabled, Python collects after the call returns: it examines the
+millions of objects just created, which takes about 0.1 s here, with or
+without threads.
+
 ## Limitations
 
 * The simdjson parser and the key and string caches are thread-local.
   `release()` frees the parser and the cached strings retained by the calling
-  thread. A parser that grows past 64 MB is freed on its own at the end of
-  that call; its caches stay. A thread that exits without `release()` leaves
-  its cached strings behind.
+  thread, and they are freed when a Python thread exits. A parser that grows
+  past 64 MB is freed on its own at the end of that call; its caches stay.
 * The module is marked free-threading compatible (`Py_MOD_GIL_NOT_USED` on
   Python 3.13 and newer), so importing it on a free-threaded build does not
-  re-enable the GIL. On a free-threaded build, `bytearray` and `memoryview`
-  inputs are copied before parsing. Subinterpreters are not supported.
+  re-enable the GIL. It is tested with many threads parsing, serializing and
+  sharing views and stream iterators on free-threaded Python 3.14 and 3.15.
+  On a free-threaded build, `bytearray` and `memoryview` inputs are copied
+  before parsing, and `dumps` and `dumpb` hold references to the items of the
+  containers they serialize, since other threads may change them.
+  Subinterpreters are not supported.

{fastsimdjson-0.3.0 → fastsimdjson-0.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "fastsimdjson"
-version = "0.3.0"
+version = "0.5.0"
 description = "Fast JSON parsing for Python, built on simdjson"
 readme = "README.md"
 authors = [

{fastsimdjson-0.3.0 → fastsimdjson-0.5.0}/src/fastsimdjson.cpp

@@ -11,6 +11,7 @@
 #include <cstring>
 #include <new>
 #include <string>
+#include <thread>
 #include <vector>
 #ifdef _WIN32
 #ifndef NOMINMAX
@@ -147,6 +148,7 @@ struct ThreadParser {
   KeyCacheEntry key_cache[KEY_CACHE_SIZE]{};
   KeyCacheEntry value_cache[KEY_CACHE_SIZE]{};
   EncodedKey *encoded_keys = nullptr; // allocated on first use by dumpb
+  bool cleanup_registered = false;    // see register_thread_cleanup
   void clear_encoded_keys() {
     if (encoded_keys != nullptr) {
       for (size_t i = 0; i < ENCODED_KEY_CACHE_SIZE; i++) {
@@ -170,6 +172,50 @@ struct ThreadParser {
 };
 thread_local ThreadParser g_thread_parser;
 
+// Frees this thread's parser, documents and caches (release()).
+void release_thread_state() {
+  delete g_thread_parser.ptr;
+  g_thread_parser.ptr = nullptr;
+  delete g_thread_parser.spare;
+  g_thread_parser.spare = nullptr;
+  clear_cache(g_thread_parser.key_cache);
+  clear_cache(g_thread_parser.value_cache);
+  g_thread_parser.clear_encoded_keys();
+}
+
+// The thread-local destructor runs after Python has deleted the thread
+// state of an exiting thread, too late to release the cached strings. So
+// each thread puts a capsule in its thread state's dict: Python clears that
+// dict while the thread state is still valid, and the capsule's destructor
+// releases the caches then. It does so only on its own thread (a thread
+// state can be cleared from another thread, at finalization).
+const char THREAD_CAPSULE[] = "fastsimdjson.thread_state";
+
+void thread_cleanup(PyObject *capsule) {
+  if (PyCapsule_GetPointer(capsule, THREAD_CAPSULE) == &g_thread_parser) {
+    g_thread_parser.cleanup_registered = false;
+    if (caches_can_decref()) {
+      release_thread_state();
+    }
+  }
+}
+
+FSJ_NOINLINE void register_thread_cleanup_slow() {
+  g_thread_parser.cleanup_registered = true;
+  PyObject *dict = PyThreadState_GetDict(); // borrowed
+  PyObject *capsule = dict ? PyCapsule_New(&g_thread_parser, THREAD_CAPSULE, thread_cleanup) : nullptr;
+  if (capsule == nullptr || PyDict_SetItemString(dict, THREAD_CAPSULE, capsule) < 0) {
+    PyErr_Clear(); // without it, the caches of an exiting thread leak, as before
+  }
+  Py_XDECREF(capsule);
+}
+
+inline void register_thread_cleanup() {
+  if (!g_thread_parser.cleanup_registered) {
+    register_thread_cleanup_slow();
+  }
+}
+
 inline uint64_t load_u64(const char *p) {
   uint64_t v;
   memcpy(&v, p, 8);
@@ -284,6 +330,7 @@ inline PyObject *make_str(const char *s, size_t len) {
 }
 
 inline void cache_store(KeyCacheEntry &entry, PyObject *u, size_t len, const KeyWords &w) {
+  register_thread_cleanup(); // the entry holds a reference: free it at thread exit
   Py_INCREF(u);
   Py_XDECREF(entry.key);
   entry.key = u;
@@ -593,6 +640,7 @@ bool ensure_parser() {
   if (g_thread_parser.ptr != nullptr) {
     return true;
   }
+  register_thread_cleanup();
   g_thread_parser.ptr = new (std::nothrow) parser();
   if (g_thread_parser.ptr == nullptr) {
     PyErr_NoMemory();
@@ -2698,6 +2746,7 @@ struct OrjsonEncoder {
       return -1;
     }
     if (keys == nullptr) {
+      register_thread_cleanup();
       keys = g_thread_parser.encoded_keys = new (std::nothrow) EncodedKey[ENCODED_KEY_CACHE_SIZE];
     }
     int rc = 0;
@@ -3004,6 +3053,9 @@ struct StreamObject {
   size_t open;     // array format: offsets of the outer '[' and ']'
   size_t close;
   PyObject *text;  // the input as str, built on the error path
+  size_t text_len; // bytes of buf that `text` covers (len, or more for a chunk)
+  bool no_fallback;   // a chunk of a parallel run: do not call json (see run_chunk)
+  bool need_fallback; // set instead of calling json, with no_fallback
   parser *p;
   document_stream *stream;
   document_stream::iterator it;
@@ -3063,7 +3115,7 @@ PyObject *stream_text(StreamObject *s) {
   if (s->text == nullptr) {
     s->text = PyUnicode_Check(s->source)
                   ? Py_NewRef(s->source)
-                  : PyUnicode_DecodeUTF8(s->buf, Py_ssize_t(s->len), nullptr);
+                  : PyUnicode_DecodeUTF8(s->buf, Py_ssize_t(s->text_len), nullptr);
   }
   return s->text;
 }
@@ -3204,6 +3256,16 @@ PyObject *stream_fallback(StreamObject *s) {
   return value;
 }
 
+// In a chunk of a parallel run, json is not called: the run continues in one
+// thread from there, where json decides as it does without threads.
+PyObject *stream_fallback_or_stop(StreamObject *s) {
+  if (s->no_fallback) {
+    s->need_fallback = true;
+    return nullptr;
+  }
+  return stream_fallback(s);
+}
+
 // The next document, or nullptr at the end or after an error.
 PyObject *stream_next_locked(StreamObject *s) {
   for (;;) {
@@ -3218,7 +3280,7 @@ PyObject *stream_next_locked(StreamObject *s) {
     if (!(s->it != s->stream->end())) {
       // Anything but separators after the last document is an incomplete
       // document: let json report it.
-      return skip_separators(s, s->last_end) < s->len ? stream_fallback(s) : nullptr;
+      return skip_separators(s, s->last_end) < s->len ? stream_fallback_or_stop(s) : nullptr;
     }
     simdjson::dom::element el;
     simdjson::error_code err = (*s->it).get(el);
@@ -3237,7 +3299,7 @@ PyObject *stream_next_locked(StreamObject *s) {
       s->active = false;
       continue;
     }
-    return stream_fallback(s);
+    return stream_fallback_or_stop(s);
   }
 }
 
@@ -3345,6 +3407,7 @@ PyObject *make_stream(PyObject *args, PyObject *kwargs, bool lazy) {
   s->active = false;
   s->base = 0;
   s->last_end = 0;
+  s->no_fallback = s->need_fallback = false;
   PyObject *self = reinterpret_cast<PyObject *>(s);
   if (must_copy) {
     PyObject *b = PyBytes_FromObject(data);
@@ -3367,7 +3430,7 @@ PyObject *make_stream(PyObject *args, PyObject *kwargs, bool lazy) {
     buf = s->copy->data();
   }
   s->buf = buf;
-  s->len = size_t(len);
+  s->len = s->text_len = size_t(len);
   // parse_many skips a UTF-8 byte order mark and, for the array format,
   // white space and the opening '['; the offsets it reports start after.
   size_t i = s->len >= 3 && memcmp(buf, "\xEF\xBB\xBF", 3) == 0 ? 3 : 0;
@@ -3403,12 +3466,410 @@ PyObject *make_stream(PyObject *args, PyObject *kwargs, bool lazy) {
   return self;
 }
 
+// ---------------------------------------------------------------------------
+// Streams over several threads (threads=N): the input is cut into chunks at
+// document boundaries (after a newline; before a record separator for
+// json_seq), and threads take the chunks in turn, each building the
+// documents of its chunk (and calling fn on them, for map_many). Without
+// the GIL (free-threaded Python), the threads run in parallel; with it, the
+// calling thread does all the work.
+//
+// The result is that of one thread: at the first chunk where anything other
+// than plain documents shows up (an error, a document that only json
+// accepts), the documents before it are kept and the calling thread goes on
+// from there as a stream without threads would. (fn may then have been
+// called on documents after that point, in other threads.)
+// ---------------------------------------------------------------------------
+
+// A stream over [from, to) of the input of `whole`, sharing its buffer.
+StreamObject *stream_over(StreamObject *whole, size_t from, size_t to) {
+  StreamObject *s = PyObject_New(StreamObject, StreamType);
+  if (s == nullptr) {
+    return nullptr;
+  }
+  s->source = Py_NewRef(whole->source);
+  s->copy = nullptr;
+  s->text = nullptr;
+  s->stream = nullptr;
+  s->p = nullptr;
+  new (&s->it) document_stream::iterator();
+  s->buf = whole->buf;
+  s->len = to;
+  s->text_len = whole->len;
+  s->batch_size = whole->batch_size;
+  s->format = whole->format;
+  s->lazy = whole->lazy;
+  s->done = false;
+  s->active = false;
+  s->base = 0;
+  s->last_end = from;
+  s->prefix = whole->prefix;
+  s->open = whole->open;
+  s->close = whole->close;
+  s->no_fallback = s->need_fallback = false;
+  s->p = new (std::nothrow) parser();
+  if (s->p == nullptr) {
+    Py_DECREF(reinterpret_cast<PyObject *>(s));
+    PyErr_NoMemory();
+    return nullptr;
+  }
+  s->p->number_as_string(true);
+  return s;
+}
+
+struct ParallelRun {
+  StreamObject *whole;
+  PyObject *fn; // borrowed, or nullptr
+  std::vector<size_t> bounds; // chunk i is [bounds[i], bounds[i + 1])
+  struct Chunk {
+    PyObject *items = nullptr; // list of the chunk's results
+    bool failed = false;       // stopped before the end of the chunk
+    PyObject *exc = nullptr;   // the exception fn raised, if it did
+    size_t resume_at = 0;      // where the one-thread run goes on
+  };
+  std::vector<Chunk> chunks;
+  std::atomic<size_t> next{0};
+  std::atomic<size_t> first_failure{SIZE_MAX};
+
+  void note_failure(size_t i) {
+    size_t f = first_failure.load();
+    while (i < f && !first_failure.compare_exchange_weak(f, i)) {
+    }
+  }
+
+  // On an attached thread.
+  void run_chunk(size_t i) {
+    Chunk &c = chunks[i];
+    c.resume_at = bounds[i];
+    StreamObject *st = stream_over(whole, bounds[i], bounds[i + 1]);
+    c.items = st ? PyList_New(0) : nullptr;
+    if (c.items == nullptr) {
+      PyErr_Clear(); // the one-thread run will report it
+      c.failed = true;
+    }
+    if (st != nullptr && c.items != nullptr) {
+      st->no_fallback = true;
+      while (first_failure.load(std::memory_order_relaxed) > i) {
+        size_t before = st->last_end;
+        PyObject *v = stream_next_locked(st);
+        if (v == nullptr) {
+          if (PyErr_Occurred() || st->need_fallback) {
+            PyErr_Clear(); // reported again by the one-thread run
+            c.failed = true;
+            c.resume_at = before;
+          }
+          break;
+        }
+        c.resume_at = st->last_end;
+        if (fn != nullptr) {
+          PyObject *r = PyObject_CallOneArg(fn, v);
+          Py_DECREF(v);
+          if (r == nullptr) {
+            PyObject *t, *val, *tb;
+            PyErr_Fetch(&t, &val, &tb);
+            PyErr_NormalizeException(&t, &val, &tb);
+            if (tb != nullptr && val != nullptr) {
+              PyException_SetTraceback(val, tb);
+            }
+            Py_XDECREF(t);
+            Py_XDECREF(tb);
+            c.exc = val;
+            c.failed = true;
+            break;
+          }
+          v = r;
+        }
+        int rc = PyList_Append(c.items, v);
+        Py_DECREF(v);
+        if (rc < 0) {
+          PyErr_Clear();
+          c.failed = true;
+          break;
+        }
+      }
+    }
+    Py_XDECREF(reinterpret_cast<PyObject *>(st));
+    if (c.failed) {
+      note_failure(i);
+    }
+  }
+
+  // Takes chunks until none is left, letting the garbage collector of
+  // other threads run between chunks.
+  void work() {
+    for (;;) {
+      size_t i = next.fetch_add(1);
+      if (i >= chunks.size() || i > first_failure.load()) {
+        return;
+      }
+      run_chunk(i);
+      Py_BEGIN_ALLOW_THREADS
+      Py_END_ALLOW_THREADS
+    }
+  }
+};
+
+// Chunk boundaries: about `n` chunks, each ending just after a newline (or
+// just before a record separator, for json_seq).
+std::vector<size_t> chunk_bounds(const StreamObject *w, size_t n) {
+  std::vector<size_t> b{0};
+  size_t step = std::max<size_t>(w->len / std::max<size_t>(n, 1), 1 << 16);
+  bool seq = w->format == simdjson::stream_format::json_sequence;
+  for (size_t at = step; at < w->len; at = b.back() + step) {
+    const void *hit = memchr(w->buf + at, seq ? '\x1e' : '\n', w->len - at);
+    if (hit == nullptr) {
+      break;
+    }
+    size_t cut = size_t(static_cast<const char *>(hit) - w->buf) + (seq ? 0 : 1);
+    if (cut >= w->len) {
+      break;
+    }
+    b.push_back(cut);
+  }
+  b.push_back(w->len);
+  return b;
+}
+
+// All the results (documents, or fn of each) of the stream `whole`, as a
+// list, using up to `threads` threads.
+PyObject *run_stream(StreamObject *whole, PyObject *fn, size_t threads) {
+  PyObject *result = PyList_New(0);
+  if (result == nullptr) {
+    return nullptr;
+  }
+  size_t resume_at = 0;
+#ifdef Py_GIL_DISABLED
+  if (threads > 1 && whole->len >= (size_t(1) << 17)) {
+    ParallelRun run;
+    run.whole = whole;
+    run.fn = fn;
+    run.bounds = chunk_bounds(whole, threads * 8);
+    run.chunks.resize(run.bounds.size() - 1);
+    // Starting a thread costs about as much as building 512 KB of documents.
+    size_t nthreads = std::min({threads, run.chunks.size(), std::max<size_t>(whole->len >> 19, 1)});
+    std::vector<std::thread> pool;
+    PyInterpreterState *interp = PyInterpreterState_Get();
+    for (size_t t = 1; t < nthreads; t++) {
+      try {
+        pool.emplace_back([&run, interp] {
+          PyThreadState *ts = PyThreadState_New(interp);
+          if (ts == nullptr) {
+            return;
+          }
+          PyEval_RestoreThread(ts);
+          run.work();
+          PyThreadState_Clear(ts);
+          PyThreadState_DeleteCurrent();
+        });
+      } catch (...) {
+        break; // fewer threads
+      }
+    }
+    run.work(); // the calling thread too
+    Py_BEGIN_ALLOW_THREADS
+    for (std::thread &t : pool) {
+      t.join();
+    }
+    Py_END_ALLOW_THREADS
+    size_t f = run.first_failure.load();
+    size_t total = 0;
+    for (size_t i = 0; i < run.chunks.size() && i <= f; i++) {
+      total += run.chunks[i].items ? size_t(PyList_GET_SIZE(run.chunks[i].items)) : 0;
+    }
+    PyObject *exc = nullptr;
+    int rc = 0;
+    if (total > 0) {
+      Py_DECREF(result);
+      result = PyList_New(Py_ssize_t(total));
+      rc = result == nullptr ? -1 : 0;
+    }
+    size_t k = 0;
+    for (size_t i = 0; i < run.chunks.size(); i++) {
+      ParallelRun::Chunk &c = run.chunks[i];
+      if (c.items != nullptr && i <= f && rc == 0) {
+        for (Py_ssize_t j = 0; j < PyList_GET_SIZE(c.items); j++) {
+          PyList_SET_ITEM(result, Py_ssize_t(k++), Py_NewRef(PyList_GET_ITEM(c.items, j)));
+        }
+      }
+      Py_XDECREF(c.items);
+      if (i == f) {
+        exc = c.exc;
+        resume_at = c.resume_at;
+      } else {
+        Py_XDECREF(c.exc);
+      }
+    }
+    if (rc < 0) {
+      Py_XDECREF(exc);
+      return nullptr;
+    }
+    if (exc != nullptr) { // fn raised: as it would have without threads
+      Py_DECREF(result);
+      PyErr_SetObject(reinterpret_cast<PyObject *>(Py_TYPE(exc)), exc);
+      Py_DECREF(exc);
+      return nullptr;
+    }
+    if (f == SIZE_MAX) {
+      return result;
+    }
+  }
+#else
+  (void)threads;
+#endif
+  // One thread, from resume_at to the end.
+  StreamObject *st = resume_at == 0 ? reinterpret_cast<StreamObject *>(
+                                          Py_NewRef(reinterpret_cast<PyObject *>(whole)))
+                                    : stream_over(whole, resume_at, whole->len);
+  if (st == nullptr) {
+    Py_DECREF(result);
+    return nullptr;
+  }
+  for (;;) {
+    PyObject *v = stream_next_locked(st);
+    if (v == nullptr) {
+      break;
+    }
+    if (fn != nullptr) {
+      PyObject *r = PyObject_CallOneArg(fn, v);
+      Py_DECREF(v);
+      if (r == nullptr) {
+        break;
+      }
+      v = r;
+    }
+    int rc = PyList_Append(result, v);
+    Py_DECREF(v);
+    if (rc < 0) {
+      break;
+    }
+  }
+  Py_DECREF(reinterpret_cast<PyObject *>(st));
+  if (PyErr_Occurred()) {
+    Py_DECREF(result);
+    return nullptr;
+  }
+  return result;
+}
+
+// Removes the keyword `threads` from kwargs (into *threads: 0 when absent or
+// None, else at least 1); returns the remaining keywords (new reference).
+PyObject *take_threads(PyObject *kwargs, Py_ssize_t *threads, bool *given) {
+  *threads = 0;
+  *given = false;
+  if (kwargs == nullptr) {
+    return nullptr;
+  }
+  PyObject *rest = PyDict_Copy(kwargs);
+  if (rest == nullptr) {
+    return nullptr;
+  }
+  PyObject *t = PyDict_GetItemString(rest, "threads"); // borrowed
+  if (t == nullptr) {
+    return rest;
+  }
+  Py_INCREF(t);
+  PyDict_DelItemString(rest, "threads");
+  *given = true;
+  if (t != Py_None) {
+    Py_ssize_t n = PyNumber_AsSsize_t(t, PyExc_OverflowError);
+    if (n == -1 && PyErr_Occurred()) {
+      Py_DECREF(t);
+      Py_DECREF(rest);
+      return nullptr;
+    }
+    if (n < 0) {
+      PyErr_SetString(PyExc_ValueError, "threads must be at least 0 (0: one per processor)");
+      Py_DECREF(t);
+      Py_DECREF(rest);
+      return nullptr;
+    }
+    *threads = n == 0 ? Py_ssize_t(std::max(1u, std::thread::hardware_concurrency())) : n;
+  } else {
+    *threads = Py_ssize_t(std::max(1u, std::thread::hardware_concurrency()));
+  }
+  Py_DECREF(t);
+  return rest;
+}
+
+// The stream for loads_many/parse_many/map_many; with threads, the list of
+// all results instead.
+PyObject *stream_or_list(PyObject *args, PyObject *kwargs, bool lazy, PyObject *fn,
+                         bool force_list) {
+  Py_ssize_t threads;
+  bool given;
+  PyObject *rest = take_threads(kwargs, &threads, &given);
+  if (rest == nullptr && PyErr_Occurred()) {
+    return nullptr;
+  }
+  PyObject *self = make_stream(args, rest, lazy);
+  Py_XDECREF(rest);
+  if (self == nullptr) {
+    return nullptr;
+  }
+  if (!given && !force_list) {
+    return self;
+  }
+  StreamObject *whole = reinterpret_cast<StreamObject *>(self);
+  using simdjson::stream_format;
+  if (given && whole->format != stream_format::whitespace_delimited &&
+      whole->format != stream_format::newline_delimited &&
+      whole->format != stream_format::json_sequence) {
+    Py_DECREF(self);
+    PyErr_SetString(PyExc_ValueError,
+                    "threads requires the format 'whitespace', 'lines' or 'json_seq'");
+    return nullptr;
+  }
+  PyObject *r = run_stream(whole, fn, given ? size_t(threads) : 1);
+  Py_DECREF(self);
+  return r;
+}
+
 PyObject *loads_many(PyObject *, PyObject *args, PyObject *kwargs) {
-  return make_stream(args, kwargs, false);
+  return stream_or_list(args, kwargs, false, nullptr, false);
 }
 
 PyObject *parse_many(PyObject *, PyObject *args, PyObject *kwargs) {
-  return make_stream(args, kwargs, true);
+  return stream_or_list(args, kwargs, true, nullptr, false);
+}
+
+// map_many(fn, data, *, format, batch_size, threads, lazy=False): [fn(doc) for
+// doc in loads_many(data)] (parse_many with lazy=True), with fn called in the
+// threads.
+PyObject *map_many(PyObject *, PyObject *args, PyObject *kwargs) {
+  if (PyTuple_GET_SIZE(args) < 1) {
+    PyErr_SetString(PyExc_TypeError, "map_many() missing required argument 'fn'");
+    return nullptr;
+  }
+  PyObject *fn = PyTuple_GET_ITEM(args, 0);
+  if (!PyCallable_Check(fn)) {
+    PyErr_SetString(PyExc_TypeError, "map_many() fn must be callable");
+    return nullptr;
+  }
+  bool lazy = false;
+  PyObject *rest = kwargs ? PyDict_Copy(kwargs) : nullptr;
+  if (kwargs != nullptr && rest == nullptr) {
+    return nullptr;
+  }
+  if (rest != nullptr) {
+    PyObject *l = PyDict_GetItemString(rest, "lazy"); // borrowed
+    if (l != nullptr) {
+      int t = PyObject_IsTrue(l);
+      if (t < 0 || PyDict_DelItemString(rest, "lazy") < 0) {
+        Py_DECREF(rest);
+        return nullptr;
+      }
+      lazy = t != 0;
+    }
+  }
+  PyObject *tail = PyTuple_GetSlice(args, 1, PyTuple_GET_SIZE(args));
+  if (tail == nullptr) {
+    Py_XDECREF(rest);
+    return nullptr;
+  }
+  PyObject *r = stream_or_list(tail, rest, lazy, fn, true);
+  Py_DECREF(tail);
+  Py_XDECREF(rest);
+  return r;
 }
 
 // Benchmarking helper: run simdjson without building Python objects.
@@ -3428,12 +3889,7 @@ PyObject *parse_only(PyObject *, PyObject *arg) {
 }
 
 PyObject *release(PyObject *, PyObject *) {
-  release_parser();
-  delete g_thread_parser.spare;
-  g_thread_parser.spare = nullptr;
-  clear_cache(g_thread_parser.key_cache);
-  clear_cache(g_thread_parser.value_cache);
-  g_thread_parser.clear_encoded_keys();
+  release_thread_state();
   Py_RETURN_NONE;
 }
 
@@ -3454,10 +3910,15 @@ PyMethodDef methods[] = {
     {"parse_file", parse_file, METH_O, "Parse the JSON file at path lazily, like parse."},
     {"loads_many", reinterpret_cast<PyCFunction>(reinterpret_cast<void (*)(void)>(loads_many)),
      METH_VARARGS | METH_KEYWORDS,
-     "Iterate over the JSON documents of data (NDJSON, JSON Lines, ...), like loads."},
+     "Iterate over the JSON documents of data (NDJSON, JSON Lines, ...), like loads; "
+     "with threads=N, the list of the documents, built by N threads."},
     {"parse_many", reinterpret_cast<PyCFunction>(reinterpret_cast<void (*)(void)>(parse_many)),
      METH_VARARGS | METH_KEYWORDS,
      "Iterate over the JSON documents of data lazily, like parse."},
+    {"map_many", reinterpret_cast<PyCFunction>(reinterpret_cast<void (*)(void)>(map_many)),
+     METH_VARARGS | METH_KEYWORDS,
+     "map_many(fn, data, *, format, batch_size, threads, lazy): the list of fn(doc) for the "
+     "documents of data, with fn called in the threads."},
     {"parse", parse, METH_O,
      "Parse JSON lazily: objects and arrays are returned as read-only views "
      "(Object, Array) whose values are converted on access."},

{fastsimdjson-0.3.0 → fastsimdjson-0.5.0/src/fastsimdjson.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fastsimdjson
-Version: 0.3.0
+Version: 0.5.0
 Summary: Fast JSON parsing for Python, built on simdjson
 Author-email: Daniel Lemire <daniel@lemire.me>
 License-Expression: Apache-2.0
@@ -197,6 +197,40 @@ returned, otherwise `JSONDecodeError` reports `json`'s message and the
 position in the whole input. A truncated last document is an error. The
 views returned by `parse_many` remain valid after the iterator moves on.
 
+### Streams on several threads: `threads=N` and `map_many`
+
+On free-threaded Python (3.14t and newer), a stream can be parsed by several
+threads at once:
+
+```python
+docs = fastsimdjson.loads_many(data, threads=16)        # a list, in order
+views = fastsimdjson.parse_many(data, threads=16)       # a list of views
+# fn runs in the threads too, on each document; its results, in order:
+scores = fastsimdjson.map_many(score, data, threads=16)
+```
+
+With `threads`, `loads_many` and `parse_many` return a list of all the
+documents instead of an iterator. `map_many(fn, data, **kw)` returns
+`[fn(doc) for doc in loads_many(data, **kw)]` (`parse_many` with
+`lazy=True`), with `fn` called in the threads. `threads=0` (or `None`) uses
+one thread per processor; small inputs use fewer threads (about one per
+512 KB). The input is cut into chunks at newlines (before each `\x1e` for
+`json_seq`), so `threads` works with the formats `"whitespace"`, `"lines"`
+and `"json_seq"`.
+
+The results, and the exceptions, are those of one thread: when a chunk
+contains an error or a document that only `json` accepts, the documents
+before it are kept and the rest is parsed in the calling thread, as without
+threads. An exception raised by `fn` is raised again, for the first document
+where it happened; `fn` may then have been called on later documents, in
+other threads, and it must be safe to call from several threads at once.
+
+`map_many` scales best: each document is built, used and freed in the same
+thread, and only `fn`'s results come back. `loads_many` must return every
+document, and the Python objects of all of them, to the calling thread.
+With the GIL (the default build of Python), the calling thread does all the
+work: `threads` has no effect on speed.
+
 ### `release`
 
 `release()` frees the simdjson parser and the string caches kept by the
@@ -433,14 +467,35 @@ msgspec's `decode_lines` and `loads_many` are on par (`loads_many` is about 2%
 faster). `parse_many` only creates the views; reading values from them adds to
 its time.
 
+### Streams on several threads
+
+300 MB of NDJSON (146,624 records: tweets, GitHub events, citm performances,
+gsoc and update-center entries), free-threaded Python 3.14.6, collector
+disabled (see below), Intel Xeon Gold 6548N (64 cores); GB/s:
+
+| threads | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| `map_many(len, data, threads=N)` | 0.91 | 1.77 | 3.28 | 6.22 | 10.9 | 15.4 | **21.2** |
+| `parse_many(data, threads=N)` | 1.82 | 3.06 | 4.80 | 6.90 | 10.2 | 12.3 | 16.4 |
+| `loads_many(data, threads=N)` | 0.82 | 1.43 | 2.26 | 3.88 | 5.77 | 7.45 | 8.63 |
+
+`map_many` is 23 times faster on 64 threads than on one. `loads_many` builds
+every document for the calling thread and is 11 times faster. With the
+collector enabled, Python collects after the call returns: it examines the
+millions of objects just created, which takes about 0.1 s here, with or
+without threads.
+
 ## Limitations
 
 * The simdjson parser and the key and string caches are thread-local.
   `release()` frees the parser and the cached strings retained by the calling
-  thread. A parser that grows past 64 MB is freed on its own at the end of
-  that call; its caches stay. A thread that exits without `release()` leaves
-  its cached strings behind.
+  thread, and they are freed when a Python thread exits. A parser that grows
+  past 64 MB is freed on its own at the end of that call; its caches stay.
 * The module is marked free-threading compatible (`Py_MOD_GIL_NOT_USED` on
   Python 3.13 and newer), so importing it on a free-threaded build does not
-  re-enable the GIL. On a free-threaded build, `bytearray` and `memoryview`
-  inputs are copied before parsing. Subinterpreters are not supported.
+  re-enable the GIL. It is tested with many threads parsing, serializing and
+  sharing views and stream iterators on free-threaded Python 3.14 and 3.15.
+  On a free-threaded build, `bytearray` and `memoryview` inputs are copied
+  before parsing, and `dumps` and `dumpb` hold references to the items of the
+  containers they serialize, since other threads may change them.
+  Subinterpreters are not supported.

{fastsimdjson-0.3.0 → fastsimdjson-0.5.0}/tests/test_dumps.py

@@ -415,3 +415,23 @@ def test_dumpb_matches_orjson_cases():
     for obj, option, _ in ORJSON_CASES:
         assert orjson_outcome(fastsimdjson.dumpb, obj, option=option) == \
             orjson_outcome(orjson.dumps, obj, option=option), (obj, option)
+
+
+def test_thread_exit_releases_caches():
+    # A key that enters dumpb's key cache (seen twice) is referenced by the
+    # cache; when the thread exits, its caches are released.
+    import sys
+    import threading
+
+    key = "".join(["thread", "-exit-", "key"])  # not interned
+    before = sys.getrefcount(key)
+
+    def work():
+        for _ in range(3):
+            assert fastsimdjson.dumpb({key: 1}) == b'{"thread-exit-key":1}'
+        fastsimdjson.loads(b'{"a": "b"}')
+
+    t = threading.Thread(target=work)
+    t.start()
+    t.join()
+    assert sys.getrefcount(key) == before

{fastsimdjson-0.3.0 → fastsimdjson-0.5.0}/tests/test_stream.py

@@ -222,3 +222,69 @@ def test_fallback_needs_the_separator():
     with pytest.raises(fastsimdjson.JSONDecodeError) as info:
         list(fastsimdjson.loads_many("[1e400 2]", format="array"))
     assert info.value.msg == "Expecting ',' delimiter"
+
+
+# threads=N: the list of all documents, as without threads. Inputs over
+# 128 KB, so that free-threaded builds use several threads.
+
+def _outcome(fn):
+    try:
+        return ("ok", fn())
+    except Exception as e:  # noqa: BLE001
+        return (type(e).__name__, str(e), getattr(e, "pos", None))
+
+
+def _threaded_input(seed, fmt):
+    r = random.Random(seed)
+    docs = []
+    for i in range(6000):
+        c = r.randrange(100)
+        if c == 0:
+            docs.append(b"[1e400]")  # json only: inf
+        elif c == 1:
+            docs.append(b'{"big": %d}' % (10**30 + i))
+        elif c == 2 and fmt == "whitespace":
+            docs.append(b'{\n  "pretty": [1,\n 2]\n}')  # spans lines
+        else:
+            docs.append(json.dumps({"i": i, "s": "x" * r.randrange(60), "f": i / 7}).encode())
+    if seed % 3 == 1:
+        docs[r.randrange(len(docs))] = b'{"broken": [1, 2'  # an error somewhere
+    if seed % 3 == 2:
+        docs[r.randrange(len(docs))] = b'"\\ud800"'  # json only: lone surrogate
+    sep = b"\n\x1e" if fmt == "json_seq" else b"\n"
+    data = sep.join(docs)
+    return b"\x1e" + data if fmt == "json_seq" else data
+
+
+@pytest.mark.parametrize("fmt", ["whitespace", "lines", "json_seq"])
+@pytest.mark.parametrize("seed", range(9))
+def test_threads_match_one_thread(fmt, seed):
+    data = _threaded_input(seed, fmt)
+    expected = _outcome(lambda: list(fastsimdjson.loads_many(data, format=fmt)))
+    for threads in (None, 0, 1, 2, 7):
+        assert _outcome(lambda: fastsimdjson.loads_many(data, format=fmt, threads=threads)) == expected
+    lazy = _outcome(lambda: [to_py(v) for v in fastsimdjson.parse_many(data, format=fmt, threads=4)])
+    assert lazy == expected
+    mapped = _outcome(lambda: fastsimdjson.map_many(json.dumps, data, format=fmt, threads=4))
+    assert mapped == (_outcome(lambda: [json.dumps(d) for d in expected[1]]) if expected[0] == "ok" else expected)
+
+
+def test_threads_api():
+    data = b"\n".join(b'{"n": %d}' % i for i in range(30000))
+    docs = fastsimdjson.loads_many(data, threads=4)
+    assert isinstance(docs, list) and docs == [{"n": i} for i in range(30000)]
+    assert fastsimdjson.map_many(lambda d: d["n"], data, threads=3) == list(range(30000))
+    assert fastsimdjson.map_many(lambda v: v["n"], data, threads=3, lazy=True) == list(range(30000))
+    assert fastsimdjson.map_many(len, b"") == []
+    with pytest.raises(ValueError):
+        fastsimdjson.loads_many(b"[1],[2]", format="comma", threads=2)
+    with pytest.raises(ValueError):
+        fastsimdjson.loads_many(data, threads=-1)
+
+    def boom(d):
+        if d["n"] == 20000:
+            raise KeyError("boom")
+        return d
+
+    with pytest.raises(KeyError, match="boom"):
+        fastsimdjson.map_many(boom, data, threads=4)