fastsimdjson 0.6.0__tar.gz → 0.7.0__tar.gz

{fastsimdjson-0.6.0/src/fastsimdjson.egg-info → fastsimdjson-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fastsimdjson
-Version: 0.6.0
+Version: 0.7.0
 Summary: Fast JSON parsing for Python, built on simdjson
 Author-email: Daniel Lemire <daniel@lemire.me>
 License-Expression: Apache-2.0
@@ -25,6 +25,7 @@ Provides-Extra: test
 Requires-Dist: pytest; extra == "test"
 Provides-Extra: bench
 Requires-Dist: orjson; extra == "bench"
+Requires-Dist: jiter; extra == "bench"
 Requires-Dist: msgspec; extra == "bench"
 Requires-Dist: pysimdjson; extra == "bench"
 Requires-Dist: cysimdjson; extra == "bench"
@@ -38,7 +39,8 @@ replacement for the `json` module's `loads`, `load`, `dumps` and `dump`, and
 it can be over 3 times faster than the standard `json.loads` and
 `json.dumps`. When you only need part of a document, its lazy `parse`
 function is faster still. It also reads streams of documents (NDJSON, JSON
-Lines). It writes JSON in two ways: `dumps` returns the same `str` as
+Lines), on many threads with free-threaded Python, and can extract only the
+fields you need from each document. It writes JSON in two ways: `dumps` returns the same `str` as
 `json.dumps`, and `dumpb` returns the same `bytes` as `orjson.dumps`, as fast
 as orjson.
 
@@ -233,6 +235,37 @@ document, and the Python objects of all of them, to the calling thread.
 With the GIL (the default build of Python), the calling thread does all the
 work: `threads` has no effect on speed.
 
+### Only some fields: `fields=`
+
+When you need a few values from each document, `fields` extracts them
+without building the documents: only the values found become Python
+objects.
+
+```python
+events = fastsimdjson.loads_many(data, fields=["/type", "/payload/action"])
+# (type, action) for each document
+
+def count(t):
+    kind, action, language = t
+    ...
+fastsimdjson.map_many(count, data, threads=64,
+                      fields=["/type", "/payload/action",
+                              "/payload/pull_request/base/repo/language"])
+```
+
+`loads_many` and `map_many` accept `fields`: a JSON Pointer (RFC 6901), or a
+list or tuple of them. Each document then becomes the value at the pointer,
+or a tuple of the values at the pointers, in order. A value that is an
+object or an array is built in full. A pointer that leads nowhere (a
+missing key, an index past the end, or a step into a scalar) gives
+`default`, `None` unless you pass another value; a JSON `null` gives `None`.
+As with `at_pointer`, a key that appears twice in an object refers to its
+first occurrence. `fields` cannot be combined with `lazy=True`.
+
+On 7.6 GB of GitHub events, `map_many` with three fields ran at 2.8 GB/s on
+one core of an Intel Xeon Gold 6548N, against 2.2 GB/s with lazy views
+(`lazy=True`). With 64 threads, it ran at 78 GB/s (lazy views: 61 GB/s).
+
 ### `release`
 
 `release()` frees the simdjson parser and the string caches kept by the
@@ -307,8 +340,8 @@ The suite builds an ~80 MB document and a list of 16,777,221 integers, so
 give it some RAM.
 
 The benchmark scripts need simdjson-data and the `bench` extra (orjson,
-msgspec, pysimdjson, cysimdjson). `bench.py` times `loads` against
-`json.loads` and orjson, `bench_lazy.py` times `parse` against pysimdjson and
+jiter, msgspec, pysimdjson, cysimdjson). `bench.py` times `loads` against
+`json.loads`, orjson and jiter, `bench_lazy.py` times `parse` against pysimdjson and
 cysimdjson, `bench_dumps.py` times `dumps` against `json.dumps` and `dumpb`
 against orjson and msgspec, and `bench_many.py` times `loads_many`.
 
@@ -347,7 +380,7 @@ geometric mean over the 22 files (higher is better).
 
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads_dark.png">
-  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads.png" width="80%" alt="Parsing JSON in Python: fastsimdjson loads 0.78 GB/s, orjson 0.60, msgspec 0.53, pysimdjson 0.44, cysimdjson 0.43, ujson 0.37, python-rapidjson 0.24, simplejson 0.23, json 0.22">
+  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads.png" width="80%" alt="Parsing JSON in Python: fastsimdjson loads 0.78 GB/s, orjson 0.60, msgspec 0.53, jiter 0.48, pysimdjson 0.44, cysimdjson 0.43, ujson 0.37, python-rapidjson 0.24, simplejson 0.23, json 0.22">
 </picture>
 
 | parser | GB/s | vs `json.loads` |
@@ -358,6 +391,7 @@ geometric mean over the 22 files (higher is better).
 | ujson 6.0.0 | 0.37 | 1.66× |
 | cysimdjson 26.27 | 0.43 | 1.96× |
 | pysimdjson 7.0.2 | 0.44 | 1.98× |
+| jiter 0.17.0 | 0.48 | 2.21× |
 | msgspec 0.22.0 | 0.53 | 2.42× |
 | orjson 3.12.0 | 0.60 | 2.74× |
 | **fastsimdjson `loads`** | **0.78** | **3.53×** |
@@ -371,7 +405,9 @@ gain comes from pausing the garbage collector while the objects are built.
 Timing each call on its own, fastsimdjson's lead over orjson is 1.33× with
 the collector enabled and 1.22× with it disabled (geometric means); without
 the collector, `canada.json`, `mesh.json` and `numbers.json` are ties. yyjson 4.0.6 is left out: it
-returns wrong strings for non-ASCII text.
+returns wrong strings for non-ASCII text. jiter (the parser of pydantic)
+caches strings by default; turning the cache off (`cache_mode=False`) makes it
+slower here (0.44 GB/s).
 
 Parsing is no longer the bottleneck. simdjson alone parses these files at
 3.1 GB/s. It accounts for about a third of the time of `loads`; the rest goes

{fastsimdjson-0.6.0 → fastsimdjson-0.7.0}/README.md

@@ -7,7 +7,8 @@ replacement for the `json` module's `loads`, `load`, `dumps` and `dump`, and
 it can be over 3 times faster than the standard `json.loads` and
 `json.dumps`. When you only need part of a document, its lazy `parse`
 function is faster still. It also reads streams of documents (NDJSON, JSON
-Lines). It writes JSON in two ways: `dumps` returns the same `str` as
+Lines), on many threads with free-threaded Python, and can extract only the
+fields you need from each document. It writes JSON in two ways: `dumps` returns the same `str` as
 `json.dumps`, and `dumpb` returns the same `bytes` as `orjson.dumps`, as fast
 as orjson.
 
@@ -202,6 +203,37 @@ document, and the Python objects of all of them, to the calling thread.
 With the GIL (the default build of Python), the calling thread does all the
 work: `threads` has no effect on speed.
 
+### Only some fields: `fields=`
+
+When you need a few values from each document, `fields` extracts them
+without building the documents: only the values found become Python
+objects.
+
+```python
+events = fastsimdjson.loads_many(data, fields=["/type", "/payload/action"])
+# (type, action) for each document
+
+def count(t):
+    kind, action, language = t
+    ...
+fastsimdjson.map_many(count, data, threads=64,
+                      fields=["/type", "/payload/action",
+                              "/payload/pull_request/base/repo/language"])
+```
+
+`loads_many` and `map_many` accept `fields`: a JSON Pointer (RFC 6901), or a
+list or tuple of them. Each document then becomes the value at the pointer,
+or a tuple of the values at the pointers, in order. A value that is an
+object or an array is built in full. A pointer that leads nowhere (a
+missing key, an index past the end, or a step into a scalar) gives
+`default`, `None` unless you pass another value; a JSON `null` gives `None`.
+As with `at_pointer`, a key that appears twice in an object refers to its
+first occurrence. `fields` cannot be combined with `lazy=True`.
+
+On 7.6 GB of GitHub events, `map_many` with three fields ran at 2.8 GB/s on
+one core of an Intel Xeon Gold 6548N, against 2.2 GB/s with lazy views
+(`lazy=True`). With 64 threads, it ran at 78 GB/s (lazy views: 61 GB/s).
+
 ### `release`
 
 `release()` frees the simdjson parser and the string caches kept by the
@@ -276,8 +308,8 @@ The suite builds an ~80 MB document and a list of 16,777,221 integers, so
 give it some RAM.
 
 The benchmark scripts need simdjson-data and the `bench` extra (orjson,
-msgspec, pysimdjson, cysimdjson). `bench.py` times `loads` against
-`json.loads` and orjson, `bench_lazy.py` times `parse` against pysimdjson and
+jiter, msgspec, pysimdjson, cysimdjson). `bench.py` times `loads` against
+`json.loads`, orjson and jiter, `bench_lazy.py` times `parse` against pysimdjson and
 cysimdjson, `bench_dumps.py` times `dumps` against `json.dumps` and `dumpb`
 against orjson and msgspec, and `bench_many.py` times `loads_many`.
 
@@ -316,7 +348,7 @@ geometric mean over the 22 files (higher is better).
 
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads_dark.png">
-  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads.png" width="80%" alt="Parsing JSON in Python: fastsimdjson loads 0.78 GB/s, orjson 0.60, msgspec 0.53, pysimdjson 0.44, cysimdjson 0.43, ujson 0.37, python-rapidjson 0.24, simplejson 0.23, json 0.22">
+  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads.png" width="80%" alt="Parsing JSON in Python: fastsimdjson loads 0.78 GB/s, orjson 0.60, msgspec 0.53, jiter 0.48, pysimdjson 0.44, cysimdjson 0.43, ujson 0.37, python-rapidjson 0.24, simplejson 0.23, json 0.22">
 </picture>
 
 | parser | GB/s | vs `json.loads` |
@@ -327,6 +359,7 @@ geometric mean over the 22 files (higher is better).
 | ujson 6.0.0 | 0.37 | 1.66× |
 | cysimdjson 26.27 | 0.43 | 1.96× |
 | pysimdjson 7.0.2 | 0.44 | 1.98× |
+| jiter 0.17.0 | 0.48 | 2.21× |
 | msgspec 0.22.0 | 0.53 | 2.42× |
 | orjson 3.12.0 | 0.60 | 2.74× |
 | **fastsimdjson `loads`** | **0.78** | **3.53×** |
@@ -340,7 +373,9 @@ gain comes from pausing the garbage collector while the objects are built.
 Timing each call on its own, fastsimdjson's lead over orjson is 1.33× with
 the collector enabled and 1.22× with it disabled (geometric means); without
 the collector, `canada.json`, `mesh.json` and `numbers.json` are ties. yyjson 4.0.6 is left out: it
-returns wrong strings for non-ASCII text.
+returns wrong strings for non-ASCII text. jiter (the parser of pydantic)
+caches strings by default; turning the cache off (`cache_mode=False`) makes it
+slower here (0.44 GB/s).
 
 Parsing is no longer the bottleneck. simdjson alone parses these files at
 3.1 GB/s. It accounts for about a third of the time of `loads`; the rest goes

{fastsimdjson-0.6.0 → fastsimdjson-0.7.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "fastsimdjson"
-version = "0.6.0"
+version = "0.7.0"
 description = "Fast JSON parsing for Python, built on simdjson"
 readme = "README.md"
 authors = [
@@ -34,7 +34,7 @@ Issues = "https://github.com/simdjson/fastpysimdjson/issues"
 
 [project.optional-dependencies]
 test = ["pytest"]
-bench = ["orjson", "msgspec", "pysimdjson", "cysimdjson"]
+bench = ["orjson", "jiter", "msgspec", "pysimdjson", "cysimdjson"]
 
 [tool.cibuildwheel]
 skip = ["pp*"]

{fastsimdjson-0.6.0 → fastsimdjson-0.7.0}/src/fastsimdjson.cpp

@@ -892,11 +892,10 @@ PyObject *value_at(DocumentObject *owner, size_t i) {
   }
 }
 
-// Tape index of the value of the first field named key, or 0.
-size_t find_key(DocumentObject *owner, size_t start, const char *key,
-                size_t len) {
-  const uint64_t *tape = tape_of(owner);
-  const uint8_t *strings = owner->doc->string_buf.get();
+// Tape index of the value of the first field named key in the object at
+// start, or 0.
+size_t find_key_in(const uint64_t *tape, const uint8_t *strings, size_t start,
+                   const char *key, size_t len) {
   size_t end = close_of(tape, start);
   for (size_t i = start + 1; i < end; i = skip(tape, i + 1)) {
     size_t idx = size_t(tape[i] & simdjson::internal::JSON_VALUE_MASK);
@@ -909,6 +908,11 @@ size_t find_key(DocumentObject *owner, size_t start, const char *key,
   return 0;
 }
 
+size_t find_key(DocumentObject *owner, size_t start, const char *key,
+                size_t len) {
+  return find_key_in(tape_of(owner), owner->doc->string_buf.get(), start, key, len);
+}
+
 // Tape index of element k of an array, starting from element j at tape
 // index i, or 0 when the array, which closes at `end`, is shorter.
 size_t walk_to(const uint64_t *tape, size_t i, size_t j, size_t k, size_t end) {
@@ -1002,6 +1006,234 @@ PyObject *at_pointer(DocumentObject *owner, size_t i, PyObject *arg) {
   return value_at(owner, i);
 }
 
+// --- fields= ------------------------------------------------------------------
+// loads_many(data, fields=...) and map_many(fn, data, fields=...): instead of
+// each document, the values at some JSON Pointers, found on the tape. Only
+// these values become Python objects.
+
+struct FieldStep {
+  std::string key;
+  size_t index;    // the token as an array index, or SIZE_MAX
+  PyObject *name;  // the token as a str (documents that json decoded)
+};
+
+struct Fields {
+  std::vector<std::vector<FieldStep>> paths;
+  std::vector<size_t> shared; // leading steps in common with the previous path
+  size_t depth = 0;           // steps of the longest path
+  bool single = false;        // fields was one str: the value, not a tuple
+  PyObject *dflt = nullptr;   // the value of a missing field
+
+  ~Fields() {
+    for (auto &p : paths) {
+      for (auto &s : p) {
+        Py_XDECREF(s.name);
+      }
+    }
+    Py_XDECREF(dflt);
+  }
+};
+
+// Appends the steps of the JSON Pointer `arg` to `path`.
+bool compile_pointer(PyObject *arg, std::vector<FieldStep> &path) {
+  Py_ssize_t plen;
+  const char *p = PyUnicode_Check(arg) ? PyUnicode_AsUTF8AndSize(arg, &plen) : nullptr;
+  if (p == nullptr) {
+    if (!PyErr_Occurred()) {
+      PyErr_Format(PyExc_TypeError, "fields must be JSON pointers (str), not %.200s",
+                   Py_TYPE(arg)->tp_name);
+    }
+    return false;
+  }
+  if (plen > 0 && p[0] != '/') {
+    PyErr_Format(PyExc_ValueError, "invalid JSON pointer: %R", arg);
+    return false;
+  }
+  const char *end = p + plen;
+  for (const char *cur = p; cur < end;) {
+    cur++; // '/'
+    const char *stop = static_cast<const char *>(memchr(cur, '/', size_t(end - cur)));
+    if (stop == nullptr) {
+      stop = end;
+    }
+    std::string token(cur, size_t(stop - cur));
+    cur = stop;
+    for (size_t t = token.find('~'); t != std::string::npos; t = token.find('~', t + 1)) {
+      if (t + 1 >= token.size() || (token[t + 1] != '0' && token[t + 1] != '1')) {
+        PyErr_Format(PyExc_ValueError, "invalid JSON pointer: %R", arg);
+        return false;
+      }
+      token.replace(t, 2, token[t + 1] == '0' ? "~" : "/");
+    }
+    bool digits = !token.empty() && (token.size() == 1 || token[0] != '0') &&
+                  token.size() <= 18;
+    size_t k = 0;
+    for (char ch : token) {
+      digits = digits && ch >= '0' && ch <= '9';
+      k = k * 10 + size_t(ch - '0');
+    }
+    PyObject *name = PyUnicode_FromStringAndSize(token.data(), Py_ssize_t(token.size()));
+    if (name == nullptr) {
+      return false;
+    }
+    path.push_back(FieldStep{std::move(token), digits ? k : SIZE_MAX, name});
+  }
+  return true;
+}
+
+// The fields of `spec` (a JSON Pointer, or a list or tuple of them), or
+// nullptr with an exception.
+Fields *compile_fields(PyObject *spec, PyObject *dflt) {
+  Fields *f = new (std::nothrow) Fields();
+  if (f == nullptr) {
+    PyErr_NoMemory();
+    return nullptr;
+  }
+  f->dflt = Py_NewRef(dflt != nullptr ? dflt : Py_None);
+  f->single = PyUnicode_Check(spec);
+  PyObject *seq = f->single ? PyTuple_Pack(1, spec)
+                  : PyList_Check(spec) || PyTuple_Check(spec)
+                      ? PySequence_Tuple(spec)
+                      : nullptr;
+  if (seq == nullptr) {
+    if (!PyErr_Occurred()) {
+      PyErr_Format(PyExc_TypeError,
+                   "fields must be a JSON pointer (str) or a list or tuple of them, not %.200s",
+                   Py_TYPE(spec)->tp_name);
+    }
+    delete f;
+    return nullptr;
+  }
+  for (Py_ssize_t n = 0; n < PyTuple_GET_SIZE(seq); n++) {
+    f->paths.emplace_back();
+    if (!compile_pointer(PyTuple_GET_ITEM(seq, n), f->paths.back())) {
+      Py_DECREF(seq);
+      delete f;
+      return nullptr;
+    }
+    const std::vector<FieldStep> &path = f->paths.back();
+    size_t common = 0;
+    if (n > 0) {
+      const std::vector<FieldStep> &prev = f->paths[size_t(n) - 1];
+      while (common < path.size() && common < prev.size() &&
+             path[common].key == prev[common].key) {
+        common++;
+      }
+    }
+    f->shared.push_back(common);
+    f->depth = std::max(f->depth, path.size());
+  }
+  Py_DECREF(seq);
+  return f;
+}
+
+// The fields of the document on the tape (root at index 1).
+PyObject *extract_fields(const document &doc, const Fields &f) {
+  const uint64_t *tape = doc.tape.get();
+  const uint8_t *strings = doc.string_buf.get();
+  Builder b(doc);
+  size_t n = f.paths.size();
+  PyObject *tuple = f.single ? nullptr : PyTuple_New(Py_ssize_t(n));
+  if (!f.single && tuple == nullptr) {
+    return nullptr;
+  }
+  // pos[d]: tape index reached after d steps of the previous path, which
+  // reached `reached` steps; paths that share a prefix walk it once.
+  size_t stack[16];
+  std::vector<size_t> heap;
+  size_t *pos = stack;
+  if (f.depth + 1 > 16) {
+    heap.resize(f.depth + 1);
+    pos = heap.data();
+  }
+  pos[0] = 1;
+  size_t reached = 0;
+  for (size_t p = 0; p < n; p++) {
+    const std::vector<FieldStep> &path = f.paths[p];
+    size_t d = std::min(f.shared[p], reached);
+    size_t i = pos[d];
+    for (; d < path.size(); d++) {
+      uint8_t type = uint8_t(tape[i] >> 56);
+      const FieldStep &s = path[d];
+      if (type == '{') {
+        i = find_key_in(tape, strings, i, s.key.data(), s.key.size());
+      } else if (type == '[' && s.index != SIZE_MAX) {
+        i = walk_to(tape, i + 1, 0, s.index, close_of(tape, i));
+      } else {
+        i = 0;
+      }
+      if (i == 0) {
+        break;
+      }
+      pos[d + 1] = i;
+    }
+    reached = d;
+    PyObject *v;
+    if (d == path.size()) {
+      v = b.build(i);
+      if (v == nullptr) {
+        Py_XDECREF(tuple);
+        return nullptr;
+      }
+    } else {
+      v = Py_NewRef(f.dflt);
+    }
+    if (f.single) {
+      return v;
+    }
+    PyTuple_SET_ITEM(tuple, Py_ssize_t(p), v);
+  }
+  return tuple;
+}
+
+PyObject *extract_fields_gc_paused(const document &doc, const Fields &f) {
+#ifndef Py_GIL_DISABLED
+  int gc_was_enabled = PyGC_Disable();
+#endif
+  PyObject *res = extract_fields(doc, f);
+#ifndef Py_GIL_DISABLED
+  if (gc_was_enabled) {
+    PyGC_Enable();
+  }
+#endif
+  return res;
+}
+
+// The fields of a document that json decoded into Python objects.
+PyObject *extract_fields_py(PyObject *value, const Fields &f) {
+  size_t n = f.paths.size();
+  PyObject *tuple = f.single ? nullptr : PyTuple_New(Py_ssize_t(n));
+  if (!f.single && tuple == nullptr) {
+    return nullptr;
+  }
+  for (size_t p = 0; p < n; p++) {
+    PyObject *v = value; // borrowed
+    for (const FieldStep &s : f.paths[p]) {
+      if (PyDict_Check(v)) {
+        v = PyDict_GetItemWithError(v, s.name);
+        if (v == nullptr && PyErr_Occurred()) {
+          Py_XDECREF(tuple);
+          return nullptr;
+        }
+      } else if (PyList_Check(v) && s.index != SIZE_MAX &&
+                 s.index < size_t(PyList_GET_SIZE(v))) {
+        v = PyList_GET_ITEM(v, Py_ssize_t(s.index));
+      } else {
+        v = nullptr;
+      }
+      if (v == nullptr) {
+        break;
+      }
+    }
+    v = Py_NewRef(v != nullptr ? v : f.dflt);
+    if (f.single) {
+      return v;
+    }
+    PyTuple_SET_ITEM(tuple, Py_ssize_t(p), v);
+  }
+  return tuple;
+}
+
 // --- Document ---------------------------------------------------------------
 
 void free_copy(document *d);
@@ -3102,6 +3334,8 @@ struct StreamObject {
   bool no_fallback;   // a chunk of a parallel run: do not call json (see run_chunk)
   bool need_fallback; // set instead of calling json, with no_fallback
   bool borrowed_parser; // p is this thread's chunk parser (not deleted)
+  Fields *fields;       // fields=: the values at these pointers, not the documents
+  bool owns_fields;     // false in the chunks of a parallel run
   parser *p;
   document_stream *stream;
   document_stream::iterator it;
@@ -3266,6 +3500,9 @@ void free_copy(document *d) {
 }
 
 PyObject *stream_value(StreamObject *s) {
+  if (s->fields != nullptr) {
+    return extract_fields_gc_paused(s->p->doc, *s->fields);
+  }
   if (!s->lazy) {
     return build_with_gc_paused(s->p->doc, 1);
   }
@@ -3313,6 +3550,11 @@ PyObject *stream_fallback(StreamObject *s) {
     return nullptr;
   }
   s->active = false; // resume with simdjson after this document
+  if (s->fields != nullptr) {
+    PyObject *v = extract_fields_py(value, *s->fields);
+    Py_DECREF(value);
+    return v;
+  }
   // As with parse(), a document that only json accepts is returned as plain
   // Python objects.
   return value;
@@ -3387,6 +3629,9 @@ void stream_dealloc(PyObject *self) {
   } else {
     delete s->p;
   }
+  if (s->owns_fields) {
+    delete s->fields;
+  }
   delete s->copy;
   Py_XDECREF(s->source);
   Py_XDECREF(s->text);
@@ -3474,6 +3719,8 @@ PyObject *make_stream(PyObject *args, PyObject *kwargs, bool lazy) {
   s->base = 0;
   s->last_end = 0;
   s->no_fallback = s->need_fallback = s->borrowed_parser = false;
+  s->fields = nullptr;
+  s->owns_fields = false;
   PyObject *self = reinterpret_cast<PyObject *>(s);
   if (must_copy) {
     PyObject *b = PyBytes_FromObject(data);
@@ -3574,6 +3821,8 @@ StreamObject *stream_over(StreamObject *whole, size_t from, size_t to, bool chun
   s->open = whole->open;
   s->close = whole->close;
   s->no_fallback = s->need_fallback = s->borrowed_parser = false;
+  s->fields = whole->fields; // whole outlives its chunks
+  s->owns_fields = false;
   if (chunk && !g_thread_parser.chunk_parser_busy) {
     if (g_thread_parser.chunk_parser == nullptr) {
       register_thread_cleanup();
@@ -4011,21 +4260,67 @@ PyObject *take_threads(PyObject *kwargs, Py_ssize_t *threads, bool *given) {
   return rest;
 }
 
+// Removes the keyword `name` from the dict kwargs (may be nullptr); returns
+// its value (new reference), or nullptr when absent or on error.
+PyObject *pop_keyword(PyObject *kwargs, const char *name) {
+  if (kwargs == nullptr) {
+    return nullptr;
+  }
+  PyObject *v = PyDict_GetItemString(kwargs, name); // borrowed
+  if (v == nullptr) {
+    return nullptr;
+  }
+  Py_INCREF(v);
+  if (PyDict_DelItemString(kwargs, name) < 0) {
+    Py_DECREF(v);
+    return nullptr;
+  }
+  return v;
+}
+
 // The stream for loads_many/parse_many/map_many; with threads, the list of
-// all results instead.
+// all results instead. With fields (not for parse_many), the values at the
+// fields' pointers come instead of the documents.
 PyObject *stream_or_list(PyObject *args, PyObject *kwargs, bool lazy, PyObject *fn,
-                         bool force_list) {
+                         bool force_list, bool allow_fields) {
   Py_ssize_t threads;
   bool given;
   PyObject *rest = take_threads(kwargs, &threads, &given);
   if (rest == nullptr && PyErr_Occurred()) {
     return nullptr;
   }
+  Fields *fields = nullptr;
+  if (allow_fields) {
+    PyObject *spec = pop_keyword(rest, "fields");
+    PyObject *dflt = PyErr_Occurred() ? nullptr : pop_keyword(rest, "default");
+    if (PyErr_Occurred()) {
+      // reported below
+    } else if (spec != nullptr && spec != Py_None) {
+      if (lazy) {
+        PyErr_SetString(PyExc_ValueError, "fields cannot be combined with lazy=True");
+      } else {
+        fields = compile_fields(spec, dflt);
+      }
+    } else if (dflt != nullptr) {
+      PyErr_SetString(PyExc_TypeError, "default requires fields");
+    }
+    Py_XDECREF(spec);
+    Py_XDECREF(dflt);
+    if (PyErr_Occurred()) {
+      Py_XDECREF(rest);
+      return nullptr;
+    }
+  }
   PyObject *self = make_stream(args, rest, lazy);
   Py_XDECREF(rest);
   if (self == nullptr) {
+    delete fields;
     return nullptr;
   }
+  if (fields != nullptr) {
+    reinterpret_cast<StreamObject *>(self)->fields = fields;
+    reinterpret_cast<StreamObject *>(self)->owns_fields = true;
+  }
   if (!given && !force_list) {
     return self;
   }
@@ -4045,16 +4340,16 @@ PyObject *stream_or_list(PyObject *args, PyObject *kwargs, bool lazy, PyObject *
 }
 
 PyObject *loads_many(PyObject *, PyObject *args, PyObject *kwargs) {
-  return stream_or_list(args, kwargs, false, nullptr, false);
+  return stream_or_list(args, kwargs, false, nullptr, false, true);
 }
 
 PyObject *parse_many(PyObject *, PyObject *args, PyObject *kwargs) {
-  return stream_or_list(args, kwargs, true, nullptr, false);
+  return stream_or_list(args, kwargs, true, nullptr, false, false);
 }
 
-// map_many(fn, data, *, format, batch_size, threads, lazy=False): [fn(doc) for
-// doc in loads_many(data)] (parse_many with lazy=True), with fn called in the
-// threads.
+// map_many(fn, data, *, format, batch_size, threads, lazy=False, fields,
+// default): [fn(doc) for doc in loads_many(data)] (parse_many with
+// lazy=True), with fn called in the threads.
 PyObject *map_many(PyObject *, PyObject *args, PyObject *kwargs) {
   if (PyTuple_GET_SIZE(args) < 1) {
     PyErr_SetString(PyExc_TypeError, "map_many() missing required argument 'fn'");
@@ -4086,7 +4381,7 @@ PyObject *map_many(PyObject *, PyObject *args, PyObject *kwargs) {
     Py_XDECREF(rest);
     return nullptr;
   }
-  PyObject *r = stream_or_list(tail, rest, lazy, fn, true);
+  PyObject *r = stream_or_list(tail, rest, lazy, fn, true, true);
   Py_DECREF(tail);
   Py_XDECREF(rest);
   return r;

{fastsimdjson-0.6.0 → fastsimdjson-0.7.0/src/fastsimdjson.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fastsimdjson
-Version: 0.6.0
+Version: 0.7.0
 Summary: Fast JSON parsing for Python, built on simdjson
 Author-email: Daniel Lemire <daniel@lemire.me>
 License-Expression: Apache-2.0
@@ -25,6 +25,7 @@ Provides-Extra: test
 Requires-Dist: pytest; extra == "test"
 Provides-Extra: bench
 Requires-Dist: orjson; extra == "bench"
+Requires-Dist: jiter; extra == "bench"
 Requires-Dist: msgspec; extra == "bench"
 Requires-Dist: pysimdjson; extra == "bench"
 Requires-Dist: cysimdjson; extra == "bench"
@@ -38,7 +39,8 @@ replacement for the `json` module's `loads`, `load`, `dumps` and `dump`, and
 it can be over 3 times faster than the standard `json.loads` and
 `json.dumps`. When you only need part of a document, its lazy `parse`
 function is faster still. It also reads streams of documents (NDJSON, JSON
-Lines). It writes JSON in two ways: `dumps` returns the same `str` as
+Lines), on many threads with free-threaded Python, and can extract only the
+fields you need from each document. It writes JSON in two ways: `dumps` returns the same `str` as
 `json.dumps`, and `dumpb` returns the same `bytes` as `orjson.dumps`, as fast
 as orjson.
 
@@ -233,6 +235,37 @@ document, and the Python objects of all of them, to the calling thread.
 With the GIL (the default build of Python), the calling thread does all the
 work: `threads` has no effect on speed.
 
+### Only some fields: `fields=`
+
+When you need a few values from each document, `fields` extracts them
+without building the documents: only the values found become Python
+objects.
+
+```python
+events = fastsimdjson.loads_many(data, fields=["/type", "/payload/action"])
+# (type, action) for each document
+
+def count(t):
+    kind, action, language = t
+    ...
+fastsimdjson.map_many(count, data, threads=64,
+                      fields=["/type", "/payload/action",
+                              "/payload/pull_request/base/repo/language"])
+```
+
+`loads_many` and `map_many` accept `fields`: a JSON Pointer (RFC 6901), or a
+list or tuple of them. Each document then becomes the value at the pointer,
+or a tuple of the values at the pointers, in order. A value that is an
+object or an array is built in full. A pointer that leads nowhere (a
+missing key, an index past the end, or a step into a scalar) gives
+`default`, `None` unless you pass another value; a JSON `null` gives `None`.
+As with `at_pointer`, a key that appears twice in an object refers to its
+first occurrence. `fields` cannot be combined with `lazy=True`.
+
+On 7.6 GB of GitHub events, `map_many` with three fields ran at 2.8 GB/s on
+one core of an Intel Xeon Gold 6548N, against 2.2 GB/s with lazy views
+(`lazy=True`). With 64 threads, it ran at 78 GB/s (lazy views: 61 GB/s).
+
 ### `release`
 
 `release()` frees the simdjson parser and the string caches kept by the
@@ -307,8 +340,8 @@ The suite builds an ~80 MB document and a list of 16,777,221 integers, so
 give it some RAM.
 
 The benchmark scripts need simdjson-data and the `bench` extra (orjson,
-msgspec, pysimdjson, cysimdjson). `bench.py` times `loads` against
-`json.loads` and orjson, `bench_lazy.py` times `parse` against pysimdjson and
+jiter, msgspec, pysimdjson, cysimdjson). `bench.py` times `loads` against
+`json.loads`, orjson and jiter, `bench_lazy.py` times `parse` against pysimdjson and
 cysimdjson, `bench_dumps.py` times `dumps` against `json.dumps` and `dumpb`
 against orjson and msgspec, and `bench_many.py` times `loads_many`.
 
@@ -347,7 +380,7 @@ geometric mean over the 22 files (higher is better).
 
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads_dark.png">
-  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads.png" width="80%" alt="Parsing JSON in Python: fastsimdjson loads 0.78 GB/s, orjson 0.60, msgspec 0.53, pysimdjson 0.44, cysimdjson 0.43, ujson 0.37, python-rapidjson 0.24, simplejson 0.23, json 0.22">
+  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads.png" width="80%" alt="Parsing JSON in Python: fastsimdjson loads 0.78 GB/s, orjson 0.60, msgspec 0.53, jiter 0.48, pysimdjson 0.44, cysimdjson 0.43, ujson 0.37, python-rapidjson 0.24, simplejson 0.23, json 0.22">
 </picture>
 
 | parser | GB/s | vs `json.loads` |
@@ -358,6 +391,7 @@ geometric mean over the 22 files (higher is better).
 | ujson 6.0.0 | 0.37 | 1.66× |
 | cysimdjson 26.27 | 0.43 | 1.96× |
 | pysimdjson 7.0.2 | 0.44 | 1.98× |
+| jiter 0.17.0 | 0.48 | 2.21× |
 | msgspec 0.22.0 | 0.53 | 2.42× |
 | orjson 3.12.0 | 0.60 | 2.74× |
 | **fastsimdjson `loads`** | **0.78** | **3.53×** |
@@ -371,7 +405,9 @@ gain comes from pausing the garbage collector while the objects are built.
 Timing each call on its own, fastsimdjson's lead over orjson is 1.33× with
 the collector enabled and 1.22× with it disabled (geometric means); without
 the collector, `canada.json`, `mesh.json` and `numbers.json` are ties. yyjson 4.0.6 is left out: it
-returns wrong strings for non-ASCII text.
+returns wrong strings for non-ASCII text. jiter (the parser of pydantic)
+caches strings by default; turning the cache off (`cache_mode=False`) makes it
+slower here (0.44 GB/s).
 
 Parsing is no longer the bottleneck. simdjson alone parses these files at
 3.1 GB/s. It accounts for about a third of the time of `loads`; the rest goes

{fastsimdjson-0.6.0 → fastsimdjson-0.7.0}/src/fastsimdjson.egg-info/requires.txt

@@ -1,6 +1,7 @@
 
 [bench]
 orjson
+jiter
 msgspec
 pysimdjson
 cysimdjson

{fastsimdjson-0.6.0 → fastsimdjson-0.7.0}/tests/test_stream.py

@@ -381,3 +381,104 @@ def test_map_many_lazy_views_kept():
         with pytest.raises(ValueError) as info:
             fastsimdjson.map_many(boom, data, threads=threads, lazy=True)
         assert to_py(info.value.args[0]) == expected[20000]
+
+
+# fields=: the values at JSON Pointers instead of the documents.
+
+_MISSING = object()
+
+
+def _pointer_get(doc, pointer, default=None):
+    if pointer == "":
+        return doc
+    for token in pointer[1:].split("/"):
+        token = token.replace("~1", "/").replace("~0", "~")
+        if isinstance(doc, dict) and token in doc:
+            doc = doc[token]
+        elif (isinstance(doc, list) and token.isdigit() and (token == "0" or token[0] != "0")
+              and int(token) < len(doc)):
+            doc = doc[int(token)]
+        else:
+            return default
+    return doc
+
+
+def _pointers(doc, prefix=""):
+    yield prefix
+    if isinstance(doc, dict):
+        for k, v in doc.items():
+            yield from _pointers(v, prefix + "/" + k.replace("~", "~0").replace("/", "~1"))
+    elif isinstance(doc, list):
+        for i, v in enumerate(doc):
+            yield from _pointers(v, f"{prefix}/{i}")
+
+
+@pytest.mark.parametrize("seed", range(30))
+def test_fields_random(seed):
+    rnd = random.Random(seed)
+    docs = [random_value(rnd) for _ in range(40)]
+    docs += [{"a/b": {"~": [1, {"k1": 2}]}, "": 3, "07": 4}, ["x", ["y"]]]
+    candidates = sorted({p for d in docs for p in _pointers(d)})
+    candidates += ["/nope", "/k1/nope", "/0/0/0", "/00", "/-", "/1x", "/a~1b/~0/1/k1", "/", "/07"]
+    text = ndjson(docs).encode()
+    for _ in range(10):
+        pointers = rnd.sample(candidates, min(len(candidates), rnd.randrange(1, 8)))
+        if rnd.random() < 0.5:
+            pointers.sort()  # shared prefixes, one after the other
+        got = list(fastsimdjson.loads_many(text, fields=pointers))
+        assert len(got) == len(docs)
+        for g, d in zip(got, docs):
+            assert isinstance(g, tuple)
+            assert all(same(a, _pointer_get(d, p)) for a, p in zip(g, pointers))
+        one = pointers[0]
+        got = list(fastsimdjson.loads_many(text, fields=one, default=_MISSING))
+        assert all(same(a, _pointer_get(d, one, _MISSING)) for a, d in zip(got, docs))
+        assert fastsimdjson.map_many(lambda t: t, text, fields=tuple(pointers)) == \
+            list(fastsimdjson.loads_many(text, fields=pointers))
+
+
+def test_fields_api():
+    data = b'{"a": 1, "b": {"c": [10, 20, {"d": "x"}]}}\n{"a": null}\n[5]\n7\n'
+    assert list(fastsimdjson.loads_many(data, fields=["/a", "/b/c/2/d", "/b/c/2", "/0"])) == [
+        (1, "x", {"d": "x"}, None), (None, None, None, None), (None, None, None, 5),
+        (None, None, None, None)]
+    # A missing field gets the default; null is a value.
+    assert list(fastsimdjson.loads_many(data, fields="/a", default=0)) == [1, None, 0, 0]
+    assert list(fastsimdjson.loads_many(data, fields="")) == list(fastsimdjson.loads_many(data))
+    assert list(fastsimdjson.loads_many(data, fields=[])) == [(), (), (), ()]
+    assert list(fastsimdjson.loads_many(data, fields=None)) == list(fastsimdjson.loads_many(data))
+    # Duplicate keys: the first one, as for views (at_pointer).
+    assert list(fastsimdjson.loads_many(b'{"a": 1, "a": 2}', fields="/a")) == [1]
+    # Documents that only json accepts.
+    assert list(fastsimdjson.loads_many(b'{"a": 1e400, "b": [1]}\n{"a": 2}',
+                                        fields=["/a", "/b/0"])) == [(float("inf"), 1), (2, None)]
+    for bad in ["a", "/~2", "/~"]:
+        with pytest.raises(ValueError):
+            fastsimdjson.loads_many(data, fields=bad)
+    for bad in [1, ["/a", 2], {"/a"}]:
+        with pytest.raises(TypeError):
+            fastsimdjson.loads_many(data, fields=bad)
+    with pytest.raises(ValueError):
+        fastsimdjson.map_many(len, data, fields="/a", lazy=True)
+    with pytest.raises(TypeError):
+        fastsimdjson.loads_many(data, default=1)
+    with pytest.raises(TypeError):
+        fastsimdjson.parse_many(data, fields="/a")
+    with pytest.raises(fastsimdjson.JSONDecodeError):
+        list(fastsimdjson.loads_many(b'{"a": 1}\n{"a": ', fields="/a"))
+
+
+@pytest.mark.parametrize("fmt", ["whitespace", "lines", "json_seq"])
+@pytest.mark.parametrize("seed", range(6))
+def test_fields_threads(fmt, seed):
+    data = _threaded_input(seed, fmt)
+    pointers = ["/i", "/s", "/big", "/0", "/pretty/1", "/f"]
+    expected = _outcome(lambda: list(fastsimdjson.loads_many(data, format=fmt, fields=pointers)))
+    if expected[0] == "ok":
+        docs = list(fastsimdjson.loads_many(data, format=fmt))
+        assert expected[1] == [tuple(_pointer_get(d, p) for p in pointers) for d in docs]
+    for threads in (None, 1, 4):
+        assert _outcome(lambda: fastsimdjson.loads_many(data, format=fmt, fields=pointers,
+                                                        threads=threads)) == expected
+        assert _outcome(lambda: fastsimdjson.map_many(lambda t: t, data, format=fmt,
+                                                      fields=pointers, threads=threads)) == expected