fastsimdjson 0.5.0__tar.gz → 0.7.0__tar.gz

{fastsimdjson-0.5.0/src/fastsimdjson.egg-info → fastsimdjson-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fastsimdjson
-Version: 0.5.0
+Version: 0.7.0
 Summary: Fast JSON parsing for Python, built on simdjson
 Author-email: Daniel Lemire <daniel@lemire.me>
 License-Expression: Apache-2.0
@@ -25,6 +25,7 @@ Provides-Extra: test
 Requires-Dist: pytest; extra == "test"
 Provides-Extra: bench
 Requires-Dist: orjson; extra == "bench"
+Requires-Dist: jiter; extra == "bench"
 Requires-Dist: msgspec; extra == "bench"
 Requires-Dist: pysimdjson; extra == "bench"
 Requires-Dist: cysimdjson; extra == "bench"
@@ -38,7 +39,8 @@ replacement for the `json` module's `loads`, `load`, `dumps` and `dump`, and
 it can be over 3 times faster than the standard `json.loads` and
 `json.dumps`. When you only need part of a document, its lazy `parse`
 function is faster still. It also reads streams of documents (NDJSON, JSON
-Lines). It writes JSON in two ways: `dumps` returns the same `str` as
+Lines), on many threads with free-threaded Python, and can extract only the
+fields you need from each document. It writes JSON in two ways: `dumps` returns the same `str` as
 `json.dumps`, and `dumpb` returns the same `bytes` as `orjson.dumps`, as fast
 as orjson.
 
@@ -225,12 +227,45 @@ threads. An exception raised by `fn` is raised again, for the first document
 where it happened; `fn` may then have been called on later documents, in
 other threads, and it must be safe to call from several threads at once.
 
-`map_many` scales best: each document is built, used and freed in the same
-thread, and only `fn`'s results come back. `loads_many` must return every
+The worker threads are started by the first call that needs them and kept
+for later calls (they wait, detached from the interpreter, between calls,
+and stop at exit). `map_many` scales best: each document is built, used and
+freed in the same thread, and only `fn`'s results come back. `loads_many` must return every
 document, and the Python objects of all of them, to the calling thread.
 With the GIL (the default build of Python), the calling thread does all the
 work: `threads` has no effect on speed.
 
+### Only some fields: `fields=`
+
+When you need a few values from each document, `fields` extracts them
+without building the documents: only the values found become Python
+objects.
+
+```python
+events = fastsimdjson.loads_many(data, fields=["/type", "/payload/action"])
+# (type, action) for each document
+
+def count(t):
+    kind, action, language = t
+    ...
+fastsimdjson.map_many(count, data, threads=64,
+                      fields=["/type", "/payload/action",
+                              "/payload/pull_request/base/repo/language"])
+```
+
+`loads_many` and `map_many` accept `fields`: a JSON Pointer (RFC 6901), or a
+list or tuple of them. Each document then becomes the value at the pointer,
+or a tuple of the values at the pointers, in order. A value that is an
+object or an array is built in full. A pointer that leads nowhere (a
+missing key, an index past the end, or a step into a scalar) gives
+`default`, `None` unless you pass another value; a JSON `null` gives `None`.
+As with `at_pointer`, a key that appears twice in an object refers to its
+first occurrence. `fields` cannot be combined with `lazy=True`.
+
+On 7.6 GB of GitHub events, `map_many` with three fields ran at 2.8 GB/s on
+one core of an Intel Xeon Gold 6548N, against 2.2 GB/s with lazy views
+(`lazy=True`). With 64 threads, it ran at 78 GB/s (lazy views: 61 GB/s).
+
 ### `release`
 
 `release()` frees the simdjson parser and the string caches kept by the
@@ -305,8 +340,8 @@ The suite builds an ~80 MB document and a list of 16,777,221 integers, so
 give it some RAM.
 
 The benchmark scripts need simdjson-data and the `bench` extra (orjson,
-msgspec, pysimdjson, cysimdjson). `bench.py` times `loads` against
-`json.loads` and orjson, `bench_lazy.py` times `parse` against pysimdjson and
+jiter, msgspec, pysimdjson, cysimdjson). `bench.py` times `loads` against
+`json.loads`, orjson and jiter, `bench_lazy.py` times `parse` against pysimdjson and
 cysimdjson, `bench_dumps.py` times `dumps` against `json.dumps` and `dumpb`
 against orjson and msgspec, and `bench_many.py` times `loads_many`.
 
@@ -345,7 +380,7 @@ geometric mean over the 22 files (higher is better).
 
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads_dark.png">
-  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads.png" width="80%" alt="Parsing JSON in Python: fastsimdjson loads 0.78 GB/s, orjson 0.60, msgspec 0.53, pysimdjson 0.44, cysimdjson 0.43, ujson 0.37, python-rapidjson 0.24, simplejson 0.23, json 0.22">
+  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads.png" width="80%" alt="Parsing JSON in Python: fastsimdjson loads 0.78 GB/s, orjson 0.60, msgspec 0.53, jiter 0.48, pysimdjson 0.44, cysimdjson 0.43, ujson 0.37, python-rapidjson 0.24, simplejson 0.23, json 0.22">
 </picture>
 
 | parser | GB/s | vs `json.loads` |
@@ -356,6 +391,7 @@ geometric mean over the 22 files (higher is better).
 | ujson 6.0.0 | 0.37 | 1.66× |
 | cysimdjson 26.27 | 0.43 | 1.96× |
 | pysimdjson 7.0.2 | 0.44 | 1.98× |
+| jiter 0.17.0 | 0.48 | 2.21× |
 | msgspec 0.22.0 | 0.53 | 2.42× |
 | orjson 3.12.0 | 0.60 | 2.74× |
 | **fastsimdjson `loads`** | **0.78** | **3.53×** |
@@ -369,7 +405,9 @@ gain comes from pausing the garbage collector while the objects are built.
 Timing each call on its own, fastsimdjson's lead over orjson is 1.33× with
 the collector enabled and 1.22× with it disabled (geometric means); without
 the collector, `canada.json`, `mesh.json` and `numbers.json` are ties. yyjson 4.0.6 is left out: it
-returns wrong strings for non-ASCII text.
+returns wrong strings for non-ASCII text. jiter (the parser of pydantic)
+caches strings by default; turning the cache off (`cache_mode=False`) makes it
+slower here (0.44 GB/s).
 
 Parsing is no longer the bottleneck. simdjson alone parses these files at
 3.1 GB/s. It accounts for about a third of the time of `loads`; the rest goes
@@ -475,15 +513,15 @@ disabled (see below), Intel Xeon Gold 6548N (64 cores); GB/s:
 
 | threads | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
 |---|---:|---:|---:|---:|---:|---:|---:|
-| `map_many(len, data, threads=N)` | 0.91 | 1.77 | 3.28 | 6.22 | 10.9 | 15.4 | **21.2** |
-| `parse_many(data, threads=N)` | 1.82 | 3.06 | 4.80 | 6.90 | 10.2 | 12.3 | 16.4 |
-| `loads_many(data, threads=N)` | 0.82 | 1.43 | 2.26 | 3.88 | 5.77 | 7.45 | 8.63 |
-
-`map_many` is 23 times faster on 64 threads than on one. `loads_many` builds
-every document for the calling thread and is 11 times faster. With the
-collector enabled, Python collects after the call returns: it examines the
-millions of objects just created, which takes about 0.1 s here, with or
-without threads.
+| `map_many(len, data, threads=N)` | 1.03 | 1.78 | 3.66 | 6.82 | 13.0 | 23.7 | **28.8** |
+| `parse_many(data, threads=N)` | 2.00 | 2.84 | 5.28 | 9.26 | 16.1 | 22.7 | 25.8 |
+| `loads_many(data, threads=N)` | 0.90 | 1.44 | 2.79 | 4.99 | 8.98 | 13.3 | 15.4 |
+
+`map_many` is 28 times faster on 64 threads than on one (up to 38 GB/s when
+nothing else runs in the process). `loads_many` builds every document for
+the calling thread and is 17 times faster. With the collector enabled,
+Python collects after the call returns: it examines the millions of objects
+just created, which takes about 0.1 s here, with or without threads.
 
 ## Limitations
 

{fastsimdjson-0.5.0 → fastsimdjson-0.7.0}/README.md

@@ -7,7 +7,8 @@ replacement for the `json` module's `loads`, `load`, `dumps` and `dump`, and
 it can be over 3 times faster than the standard `json.loads` and
 `json.dumps`. When you only need part of a document, its lazy `parse`
 function is faster still. It also reads streams of documents (NDJSON, JSON
-Lines). It writes JSON in two ways: `dumps` returns the same `str` as
+Lines), on many threads with free-threaded Python, and can extract only the
+fields you need from each document. It writes JSON in two ways: `dumps` returns the same `str` as
 `json.dumps`, and `dumpb` returns the same `bytes` as `orjson.dumps`, as fast
 as orjson.
 
@@ -194,12 +195,45 @@ threads. An exception raised by `fn` is raised again, for the first document
 where it happened; `fn` may then have been called on later documents, in
 other threads, and it must be safe to call from several threads at once.
 
-`map_many` scales best: each document is built, used and freed in the same
-thread, and only `fn`'s results come back. `loads_many` must return every
+The worker threads are started by the first call that needs them and kept
+for later calls (they wait, detached from the interpreter, between calls,
+and stop at exit). `map_many` scales best: each document is built, used and
+freed in the same thread, and only `fn`'s results come back. `loads_many` must return every
 document, and the Python objects of all of them, to the calling thread.
 With the GIL (the default build of Python), the calling thread does all the
 work: `threads` has no effect on speed.
 
+### Only some fields: `fields=`
+
+When you need a few values from each document, `fields` extracts them
+without building the documents: only the values found become Python
+objects.
+
+```python
+events = fastsimdjson.loads_many(data, fields=["/type", "/payload/action"])
+# (type, action) for each document
+
+def count(t):
+    kind, action, language = t
+    ...
+fastsimdjson.map_many(count, data, threads=64,
+                      fields=["/type", "/payload/action",
+                              "/payload/pull_request/base/repo/language"])
+```
+
+`loads_many` and `map_many` accept `fields`: a JSON Pointer (RFC 6901), or a
+list or tuple of them. Each document then becomes the value at the pointer,
+or a tuple of the values at the pointers, in order. A value that is an
+object or an array is built in full. A pointer that leads nowhere (a
+missing key, an index past the end, or a step into a scalar) gives
+`default`, `None` unless you pass another value; a JSON `null` gives `None`.
+As with `at_pointer`, a key that appears twice in an object refers to its
+first occurrence. `fields` cannot be combined with `lazy=True`.
+
+On 7.6 GB of GitHub events, `map_many` with three fields ran at 2.8 GB/s on
+one core of an Intel Xeon Gold 6548N, against 2.2 GB/s with lazy views
+(`lazy=True`). With 64 threads, it ran at 78 GB/s (lazy views: 61 GB/s).
+
 ### `release`
 
 `release()` frees the simdjson parser and the string caches kept by the
@@ -274,8 +308,8 @@ The suite builds an ~80 MB document and a list of 16,777,221 integers, so
 give it some RAM.
 
 The benchmark scripts need simdjson-data and the `bench` extra (orjson,
-msgspec, pysimdjson, cysimdjson). `bench.py` times `loads` against
-`json.loads` and orjson, `bench_lazy.py` times `parse` against pysimdjson and
+jiter, msgspec, pysimdjson, cysimdjson). `bench.py` times `loads` against
+`json.loads`, orjson and jiter, `bench_lazy.py` times `parse` against pysimdjson and
 cysimdjson, `bench_dumps.py` times `dumps` against `json.dumps` and `dumpb`
 against orjson and msgspec, and `bench_many.py` times `loads_many`.
 
@@ -314,7 +348,7 @@ geometric mean over the 22 files (higher is better).
 
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads_dark.png">
-  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads.png" width="80%" alt="Parsing JSON in Python: fastsimdjson loads 0.78 GB/s, orjson 0.60, msgspec 0.53, pysimdjson 0.44, cysimdjson 0.43, ujson 0.37, python-rapidjson 0.24, simplejson 0.23, json 0.22">
+  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads.png" width="80%" alt="Parsing JSON in Python: fastsimdjson loads 0.78 GB/s, orjson 0.60, msgspec 0.53, jiter 0.48, pysimdjson 0.44, cysimdjson 0.43, ujson 0.37, python-rapidjson 0.24, simplejson 0.23, json 0.22">
 </picture>
 
 | parser | GB/s | vs `json.loads` |
@@ -325,6 +359,7 @@ geometric mean over the 22 files (higher is better).
 | ujson 6.0.0 | 0.37 | 1.66× |
 | cysimdjson 26.27 | 0.43 | 1.96× |
 | pysimdjson 7.0.2 | 0.44 | 1.98× |
+| jiter 0.17.0 | 0.48 | 2.21× |
 | msgspec 0.22.0 | 0.53 | 2.42× |
 | orjson 3.12.0 | 0.60 | 2.74× |
 | **fastsimdjson `loads`** | **0.78** | **3.53×** |
@@ -338,7 +373,9 @@ gain comes from pausing the garbage collector while the objects are built.
 Timing each call on its own, fastsimdjson's lead over orjson is 1.33× with
 the collector enabled and 1.22× with it disabled (geometric means); without
 the collector, `canada.json`, `mesh.json` and `numbers.json` are ties. yyjson 4.0.6 is left out: it
-returns wrong strings for non-ASCII text.
+returns wrong strings for non-ASCII text. jiter (the parser of pydantic)
+caches strings by default; turning the cache off (`cache_mode=False`) makes it
+slower here (0.44 GB/s).
 
 Parsing is no longer the bottleneck. simdjson alone parses these files at
 3.1 GB/s. It accounts for about a third of the time of `loads`; the rest goes
@@ -444,15 +481,15 @@ disabled (see below), Intel Xeon Gold 6548N (64 cores); GB/s:
 
 | threads | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
 |---|---:|---:|---:|---:|---:|---:|---:|
-| `map_many(len, data, threads=N)` | 0.91 | 1.77 | 3.28 | 6.22 | 10.9 | 15.4 | **21.2** |
-| `parse_many(data, threads=N)` | 1.82 | 3.06 | 4.80 | 6.90 | 10.2 | 12.3 | 16.4 |
-| `loads_many(data, threads=N)` | 0.82 | 1.43 | 2.26 | 3.88 | 5.77 | 7.45 | 8.63 |
-
-`map_many` is 23 times faster on 64 threads than on one. `loads_many` builds
-every document for the calling thread and is 11 times faster. With the
-collector enabled, Python collects after the call returns: it examines the
-millions of objects just created, which takes about 0.1 s here, with or
-without threads.
+| `map_many(len, data, threads=N)` | 1.03 | 1.78 | 3.66 | 6.82 | 13.0 | 23.7 | **28.8** |
+| `parse_many(data, threads=N)` | 2.00 | 2.84 | 5.28 | 9.26 | 16.1 | 22.7 | 25.8 |
+| `loads_many(data, threads=N)` | 0.90 | 1.44 | 2.79 | 4.99 | 8.98 | 13.3 | 15.4 |
+
+`map_many` is 28 times faster on 64 threads than on one (up to 38 GB/s when
+nothing else runs in the process). `loads_many` builds every document for
+the calling thread and is 17 times faster. With the collector enabled,
+Python collects after the call returns: it examines the millions of objects
+just created, which takes about 0.1 s here, with or without threads.
 
 ## Limitations
 

{fastsimdjson-0.5.0 → fastsimdjson-0.7.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "fastsimdjson"
-version = "0.5.0"
+version = "0.7.0"
 description = "Fast JSON parsing for Python, built on simdjson"
 readme = "README.md"
 authors = [
@@ -34,7 +34,7 @@ Issues = "https://github.com/simdjson/fastpysimdjson/issues"
 
 [project.optional-dependencies]
 test = ["pytest"]
-bench = ["orjson", "msgspec", "pysimdjson", "cysimdjson"]
+bench = ["orjson", "jiter", "msgspec", "pysimdjson", "cysimdjson"]
 
 [tool.cibuildwheel]
 skip = ["pp*"]