json-stream 2.1.0__tar.gz → 2.4.1__tar.gz

{json-stream-2.1.0/src/json_stream.egg-info → json_stream-2.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: json-stream
-Version: 2.1.0
+Version: 2.4.1
 Summary: Streaming JSON encoder and decoder
 Author-email: Jamie Cockburn <jamie_cockburn@hotmail.co.uk>
 License: Copyright (c) 2020 Jamie Cockburn
@@ -33,20 +33,27 @@ Classifier: Intended Audience :: Developers
 Classifier: Topic :: Software Development :: Libraries
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3 :: Only
-Classifier: Programming Language :: Python :: 3.7
 Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
 Requires-Python: <4,>=3.5
 Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: json-stream-rs-tokenizer>=0.4.17
 Provides-Extra: requests
+Requires-Dist: requests; extra == "requests"
 Provides-Extra: httpx
-License-File: LICENSE.txt
+Requires-Dist: httpx; extra == "httpx"
+Dynamic: license-file
 
 # json-stream
 
 [![Tests](https://github.com/daggaz/json-stream/actions/workflows/tests.yml/badge.svg)](https://github.com/daggaz/json-stream/actions/workflows/tests.yml)
+[![codecov](https://codecov.io/gh/daggaz/json-stream/graph/badge.svg?token=SWWYF2KHMC)](https://codecov.io/gh/daggaz/json-stream)
 [![PyPI package and version badge](https://img.shields.io/pypi/v/json-stream)](https://pypi.org/project/json-stream)
 [![Supported Python versions badge](https://img.shields.io/pypi/pyversions/json-stream)](https://pypi.org/project/json-stream/)
 [![Donate](https://img.shields.io/badge/buy%20me%20a%20coffee-donate-blue.svg)](https://www.buymeacoffee.com/daggaz)
@@ -55,8 +62,8 @@ Simple streaming JSON parser and encoder.
 
 When [reading](#reading) JSON data, `json-stream` can decode JSON data in 
 a streaming manner, providing a pythonic dict/list-like interface, or a
-[visitor-based interfeace](#visitor). Can stream from files, [URLs](#urls) 
-or [iterators](#iterators).
+[visitor-based interface](#visitor). It can stream from files, [URLs](#urls) 
+or [iterators](#iterators). It can process [multiple JSON documents](#multiple) in a single stream.
 
 When [writing](#writing) JSON data, `json-stream` can stream JSON objects 
 as you generate them.
@@ -109,8 +116,8 @@ It is also possible to "mix" the modes as you consume the data.
 
 This mode is appropriate if you can consume the data iteratively. You cannot 
 move backwards through the stream to read data that has already been skipped
-over. It is the mode you **must** use if you want process large amounts of
-JSON data without consuming large amounts of memory required.
+over. It is the mode you **must** use if you want to process large amounts of
+JSON data without consuming large amounts of memory.
 
 In transient mode, only the data currently being read is stored in memory. Any
 data previously read from the stream is discarded (it's up to you what to do 
@@ -272,6 +279,65 @@ z at path ('xxxx', 3)
 [] at path ('xxxx', 5)
 ```
 
+### <a id="multiple"></a> Multiple JSON documents: `load_many()` and `visit_many()`
+
+Sometimes JSON data arrives as a sequence of top‑level JSON texts rather than a single array/object. json-stream supports this pattern with:
+- `json_stream.load_many(...)`: yields each top-level JSON value as it is parsed.
+- `json_stream.visit_many(...)`: visits each top-level JSON value and yields control after each one.
+
+These functions are useful for common streaming formats:
+- NDJSON (Newline-Delimited JSON, also known as JSON Lines, content-type often application/x-ndjson): one JSON value per line, separated by a single "\n".
+- Concatenated or sequential JSON documents: complete JSON texts written back-to-back without delimiters.
+
+Note about concatenated JSON without explicit delimiters:
+
+When multiple top‑level JSON documents are simply concatenated with 
+no delimiters (no newlines or no spaces), json-stream can unambiguously 
+detect the document boundaries for the following top-level values:
+- objects: `{ ... }`
+- arrays: `[ ... ]`
+- literals: `true`, `false`, `null`
+
+However, numbers and strings require a delimiter between consecutive
+documents so the tokenizer can tell where one ends and the next begins.
+A delimiter can be as simple as a single whitespace or a newline. 
+
+Examples:
+- Valid without delimiters (will parse as 2 docs):
+  - `{"a":1}{"b":2}` → `{...}`, `{...}`
+  - `[1][2]` → `[1]`, `[2]`
+  - `truefalse` → `true`, `false`
+  - `null[]` → `null`, `[]`
+- These will error without a delimiter between documents (e.g. a space or newline):
+  - `3true`
+  - `0"x"`
+  - `"hi""there"
+
+#### Examples
+
+Read many from a file containing NDJSON or concatenated JSON:
+```python
+import json_stream
+
+with open("events.ndjson", "rb") as f:  # NDJSON: one JSON object per line
+    for item in json_stream.load_many(f):
+        # each item is the top-level item in each JSON text
+        handle(item)
+```
+
+Visit many values in many documents:
+```python
+import json_stream
+
+def visitor(value, path):
+    # called depth-first for each value within the current top-level JSON text
+    ...
+
+for _ in json_stream.visit_many(open("events.ndjson", "rb"), visitor):
+    # loop advances one top-level JSON text at a time
+    pass
+```
+
 ### <a id="urls"></a> Stream a URL
 
 `json_stream` knows how to stream directly from a URL using a variety of packages.
@@ -378,8 +444,7 @@ with httpx.Client() as client, client.stream('GET', 'http://example.com/data.jso
 
 ### <a id="iterators"></a> Stream an iterable
 
-`json-stream`'s parsing functions can take any iterable object that produces encoded JSON as
-`byte` objects.
+`json-stream`'s parsing functions can take any iterable that produces encoded JSON chunks. The chunks can be `byte`s or `str`s.
 
 ```python
 import json_stream
@@ -432,6 +497,21 @@ with JSONStreamEncoder():
     some_library_function_out_of_your_control(data)
 ```
 
+### Converting to standard Python types
+
+To convert a json-stream `dict`-like or `list`-like object and all its
+descendants to a standard `list` and `dict`, you can use the
+`json_stream.to_standard_types` utility:
+
+```python
+# JSON: {"round": 1, "results": [1, 2, 3]}
+data = json_stream.load(f)
+results = data["results"]
+print(results)  # prints <TransientStreamingJSONList: TRANSIENT, STREAMING>
+converted = json_stream.to_standard_types(results)
+print(converted)  # prints [1, 2, 3]
+```
+
 #### Thread safety (experimental)
 
 There is also a thread-safe version of the `json.dump` context manager:
@@ -682,23 +762,28 @@ parsing a simple JSON list as a stream.
 
 ## Yajl-Py
 
-[Yajl-Py](https://pykler.github.io/yajl-py/) is a wrapper around the Yajl JSON library that can be used to 
+[Yajl-Py](https://pykler.github.io/yajl-py/) is a wrapper around the C YAJL JSON library that can be used to 
 generate SAX style events while parsing JSON.
 
 ### Why not Yajl-Py?
 
-* It's not pure python
+* No pure python implementation
 * It does not provide a pythonic `dict`/`list`-like interface 
 
-# Build
-```bash
-cd ~/sources/json-stream/
-python3 -m venv ~/build/
-. ~/build/bin/activate
-pip install --upgrade build twine
-python -m build
-twine upload dist/*
-```
+## jsonslicer
+
+[jsonslicer](https://github.com/AMDmi3/jsonslicer) is another wrapper around the YAJL C library with a
+path lookup based interface.
+
+### Why not jsonslicer?
+
+* No pure python implementation
+* It does not provide a pythonic `dict`/`list`-like interface
+* Must know all data paths lookup in advance (or make multiple passes)
+
+# Contributing
+
+See the project [contribution guide](https://github.com/daggaz/json-stream/blob/master/CONTRIBUTING.md).
 
 # Donations
 

{json-stream-2.1.0 → json_stream-2.4.1}/README.md

@@ -1,6 +1,7 @@
 # json-stream
 
 [![Tests](https://github.com/daggaz/json-stream/actions/workflows/tests.yml/badge.svg)](https://github.com/daggaz/json-stream/actions/workflows/tests.yml)
+[![codecov](https://codecov.io/gh/daggaz/json-stream/graph/badge.svg?token=SWWYF2KHMC)](https://codecov.io/gh/daggaz/json-stream)
 [![PyPI package and version badge](https://img.shields.io/pypi/v/json-stream)](https://pypi.org/project/json-stream)
 [![Supported Python versions badge](https://img.shields.io/pypi/pyversions/json-stream)](https://pypi.org/project/json-stream/)
 [![Donate](https://img.shields.io/badge/buy%20me%20a%20coffee-donate-blue.svg)](https://www.buymeacoffee.com/daggaz)
@@ -9,8 +10,8 @@ Simple streaming JSON parser and encoder.
 
 When [reading](#reading) JSON data, `json-stream` can decode JSON data in 
 a streaming manner, providing a pythonic dict/list-like interface, or a
-[visitor-based interfeace](#visitor). Can stream from files, [URLs](#urls) 
-or [iterators](#iterators).
+[visitor-based interface](#visitor). It can stream from files, [URLs](#urls) 
+or [iterators](#iterators). It can process [multiple JSON documents](#multiple) in a single stream.
 
 When [writing](#writing) JSON data, `json-stream` can stream JSON objects 
 as you generate them.
@@ -63,8 +64,8 @@ It is also possible to "mix" the modes as you consume the data.
 
 This mode is appropriate if you can consume the data iteratively. You cannot 
 move backwards through the stream to read data that has already been skipped
-over. It is the mode you **must** use if you want process large amounts of
-JSON data without consuming large amounts of memory required.
+over. It is the mode you **must** use if you want to process large amounts of
+JSON data without consuming large amounts of memory.
 
 In transient mode, only the data currently being read is stored in memory. Any
 data previously read from the stream is discarded (it's up to you what to do 
@@ -226,6 +227,65 @@ z at path ('xxxx', 3)
 [] at path ('xxxx', 5)
 ```
 
+### <a id="multiple"></a> Multiple JSON documents: `load_many()` and `visit_many()`
+
+Sometimes JSON data arrives as a sequence of top‑level JSON texts rather than a single array/object. json-stream supports this pattern with:
+- `json_stream.load_many(...)`: yields each top-level JSON value as it is parsed.
+- `json_stream.visit_many(...)`: visits each top-level JSON value and yields control after each one.
+
+These functions are useful for common streaming formats:
+- NDJSON (Newline-Delimited JSON, also known as JSON Lines, content-type often application/x-ndjson): one JSON value per line, separated by a single "\n".
+- Concatenated or sequential JSON documents: complete JSON texts written back-to-back without delimiters.
+
+Note about concatenated JSON without explicit delimiters:
+
+When multiple top‑level JSON documents are simply concatenated with 
+no delimiters (no newlines or no spaces), json-stream can unambiguously 
+detect the document boundaries for the following top-level values:
+- objects: `{ ... }`
+- arrays: `[ ... ]`
+- literals: `true`, `false`, `null`
+
+However, numbers and strings require a delimiter between consecutive
+documents so the tokenizer can tell where one ends and the next begins.
+A delimiter can be as simple as a single whitespace or a newline. 
+
+Examples:
+- Valid without delimiters (will parse as 2 docs):
+  - `{"a":1}{"b":2}` → `{...}`, `{...}`
+  - `[1][2]` → `[1]`, `[2]`
+  - `truefalse` → `true`, `false`
+  - `null[]` → `null`, `[]`
+- These will error without a delimiter between documents (e.g. a space or newline):
+  - `3true`
+  - `0"x"`
+  - `"hi""there"
+
+#### Examples
+
+Read many from a file containing NDJSON or concatenated JSON:
+```python
+import json_stream
+
+with open("events.ndjson", "rb") as f:  # NDJSON: one JSON object per line
+    for item in json_stream.load_many(f):
+        # each item is the top-level item in each JSON text
+        handle(item)
+```
+
+Visit many values in many documents:
+```python
+import json_stream
+
+def visitor(value, path):
+    # called depth-first for each value within the current top-level JSON text
+    ...
+
+for _ in json_stream.visit_many(open("events.ndjson", "rb"), visitor):
+    # loop advances one top-level JSON text at a time
+    pass
+```
+
 ### <a id="urls"></a> Stream a URL
 
 `json_stream` knows how to stream directly from a URL using a variety of packages.
@@ -332,8 +392,7 @@ with httpx.Client() as client, client.stream('GET', 'http://example.com/data.jso
 
 ### <a id="iterators"></a> Stream an iterable
 
-`json-stream`'s parsing functions can take any iterable object that produces encoded JSON as
-`byte` objects.
+`json-stream`'s parsing functions can take any iterable that produces encoded JSON chunks. The chunks can be `byte`s or `str`s.
 
 ```python
 import json_stream
@@ -386,6 +445,21 @@ with JSONStreamEncoder():
     some_library_function_out_of_your_control(data)
 ```
 
+### Converting to standard Python types
+
+To convert a json-stream `dict`-like or `list`-like object and all its
+descendants to a standard `list` and `dict`, you can use the
+`json_stream.to_standard_types` utility:
+
+```python
+# JSON: {"round": 1, "results": [1, 2, 3]}
+data = json_stream.load(f)
+results = data["results"]
+print(results)  # prints <TransientStreamingJSONList: TRANSIENT, STREAMING>
+converted = json_stream.to_standard_types(results)
+print(converted)  # prints [1, 2, 3]
+```
+
 #### Thread safety (experimental)
 
 There is also a thread-safe version of the `json.dump` context manager:
@@ -636,23 +710,28 @@ parsing a simple JSON list as a stream.
 
 ## Yajl-Py
 
-[Yajl-Py](https://pykler.github.io/yajl-py/) is a wrapper around the Yajl JSON library that can be used to 
+[Yajl-Py](https://pykler.github.io/yajl-py/) is a wrapper around the C YAJL JSON library that can be used to 
 generate SAX style events while parsing JSON.
 
 ### Why not Yajl-Py?
 
-* It's not pure python
+* No pure python implementation
 * It does not provide a pythonic `dict`/`list`-like interface 
 
-# Build
-```bash
-cd ~/sources/json-stream/
-python3 -m venv ~/build/
-. ~/build/bin/activate
-pip install --upgrade build twine
-python -m build
-twine upload dist/*
-```
+## jsonslicer
+
+[jsonslicer](https://github.com/AMDmi3/jsonslicer) is another wrapper around the YAJL C library with a
+path lookup based interface.
+
+### Why not jsonslicer?
+
+* No pure python implementation
+* It does not provide a pythonic `dict`/`list`-like interface
+* Must know all data paths lookup in advance (or make multiple passes)
+
+# Contributing
+
+See the project [contribution guide](https://github.com/daggaz/json-stream/blob/master/CONTRIBUTING.md).
 
 # Donations
 

{json-stream-2.1.0 → json_stream-2.4.1}/pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "json-stream"
-version = "2.1.0"
+version = "2.4.1"
 authors = [
     {name = "Jamie Cockburn", email="jamie_cockburn@hotmail.co.uk"},
 ]
@@ -18,14 +18,16 @@ classifiers = [
     "Topic :: Software Development :: Libraries",
     "License :: OSI Approved :: MIT License",
     "Programming Language :: Python :: 3 :: Only",
-    "Programming Language :: Python :: 3.7",
     "Programming Language :: Python :: 3.8",
     "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
 ]
 requires-python = ">=3.5,<4"
-dependencies = ["json-stream-rs-tokenizer >= 0.4.8"]
+dependencies = ["json-stream-rs-tokenizer >= 0.4.17"]
 dynamic = ["readme"]
 
 [project.urls]

json_stream-2.4.1/src/json_stream/__init__.py

@@ -0,0 +1,4 @@
+from json_stream.loader import load, load_many  # noqa: F401
+from json_stream.visitor import visit, visit_many  # noqa: F401
+from json_stream.writer import streamable_list, streamable_dict  # noqa: F401
+from json_stream.util import to_standard_types

{json-stream-2.1.0 → json_stream-2.4.1}/src/json_stream/base.py

@@ -3,7 +3,7 @@ import copy
 from abc import ABC
 from collections import OrderedDict
 from itertools import chain
-from typing import Optional, Iterator, Any
+from typing import Optional, Iterator, Any, Mapping, Sequence
 
 from json_stream.tokenizer import TokenType
 
@@ -13,6 +13,8 @@ class TransientAccessException(Exception):
 
 
 class StreamingJSONBase(ABC):
+    INCOMPLETE_ERROR = "Unexpected end of file"
+
     @classmethod
     def factory(cls, token, token_stream, persistent):
         if persistent:
@@ -47,6 +49,8 @@ class StreamingJSONBase(ABC):
             try:
                 item = self._load_item()
             except StopIteration:
+                if self.streaming:
+                    raise ValueError(self.INCOMPLETE_ERROR)
                 return
             yield item
 
@@ -134,6 +138,8 @@ class TransientStreamingJSONBase(StreamingJSONBase, ABC):
 
 
 class StreamingJSONList(StreamingJSONBase, ABC):
+    INCOMPLETE_ERROR = "Unterminated list at end of file"
+
     def _load_item(self):
         token_type, v = next(self._stream)
         if token_type == TokenType.OPERATOR:
@@ -153,6 +159,7 @@ class StreamingJSONList(StreamingJSONBase, ABC):
         return self._iter_items()
 
 
+@Sequence.register
 class PersistentStreamingJSONList(PersistentStreamingJSONBase, StreamingJSONList):
     def _init_persistent_data(self):
         return []
@@ -178,6 +185,7 @@ class PersistentStreamingJSONList(PersistentStreamingJSONBase, StreamingJSONList
         return self._find_item(k)
 
 
+@Sequence.register
 class TransientStreamingJSONList(TransientStreamingJSONBase, StreamingJSONList):
     def __init__(self, token_stream):
         super().__init__(token_stream)
@@ -198,6 +206,8 @@ class TransientStreamingJSONList(TransientStreamingJSONBase, StreamingJSONList):
 
 
 class StreamingJSONObject(StreamingJSONBase, ABC):
+    INCOMPLETE_ERROR = "Unterminated object at end of file"
+
     def _load_item(self):
         token_type, k = next(self._stream)
         if token_type == TokenType.OPERATOR:
@@ -242,6 +252,7 @@ class StreamingJSONObject(StreamingJSONBase, ABC):
             return default
 
 
+@Mapping.register
 class PersistentStreamingJSONObject(PersistentStreamingJSONBase, StreamingJSONObject):
     def _init_persistent_data(self):
         return OrderedDict()
@@ -268,6 +279,7 @@ class PersistentStreamingJSONObject(PersistentStreamingJSONBase, StreamingJSONOb
         return self._find_item(k)
 
 
+@Mapping.register
 class TransientStreamingJSONObject(TransientStreamingJSONBase, StreamingJSONObject):
     def _find_item(self, k):
         was_started = self._started

{json-stream-2.1.0 → json_stream-2.4.1}/src/json_stream/dump/__init__.py

@@ -1,16 +1,15 @@
 import json
 
-from json_stream.base import StreamingJSONObject, StreamingJSONList
+from json_stream import to_standard_types
+from json_stream.base import StreamingJSONBase
 
 _original_default = json.JSONEncoder().default
 
 
 class JSONStreamEncoder(json.JSONEncoder):
     def default(self, obj):
-        if isinstance(obj, StreamingJSONObject):
-            return dict(obj)
-        if isinstance(obj, StreamingJSONList):
-            return list(obj)
+        if isinstance(obj, StreamingJSONBase):
+            return to_standard_types(obj)
         return _original_default(obj)
 
     def __enter__(self):

{json-stream-2.1.0 → json_stream-2.4.1}/src/json_stream/dump/tests/test_dump.py

@@ -10,16 +10,32 @@ class TestDump(TestCase):
     JSON = '{"count": 3, "results": ["a", "b", "c"]}'
 
     def test_dump_default(self):
-        data = json_stream.load(StringIO(self.JSON), persistent=True)
+        data = json_stream.load(StringIO(self.JSON))
         output = json.dumps(data, default=default)
         self._assert_json_okay(output)
 
     def test_dump_cls(self):
-        data = json_stream.load(StringIO(self.JSON), persistent=True)
+        data = json_stream.load(StringIO(self.JSON))
         output = json.dumps(data, cls=JSONStreamEncoder)
         self._assert_json_okay(output)
 
     def test_dump_context(self):
+        data = json_stream.load(StringIO(self.JSON))
+        with JSONStreamEncoder():
+            output = json.dumps(data)
+        self._assert_json_okay(output)
+
+    def test_dump_default_persistent(self):
+        data = json_stream.load(StringIO(self.JSON), persistent=True)
+        output = json.dumps(data, default=default)
+        self._assert_json_okay(output)
+
+    def test_dump_cls_persistent(self):
+        data = json_stream.load(StringIO(self.JSON), persistent=True)
+        output = json.dumps(data, cls=JSONStreamEncoder)
+        self._assert_json_okay(output)
+
+    def test_dump_context_persistent(self):
         data = json_stream.load(StringIO(self.JSON), persistent=True)
         with JSONStreamEncoder():
             output = json.dumps(data)

{json-stream-2.1.0 → json_stream-2.4.1}/src/json_stream/httpx/__init__.py

@@ -13,5 +13,13 @@ def load(response, persistent=False, tokenizer=default_tokenizer, chunk_size=CON
     return json_stream.load(_to_iterable(response, chunk_size), persistent=persistent, tokenizer=tokenizer)
 
 
+def load_many(response, persistent=False, tokenizer=default_tokenizer, chunk_size=CONTENT_CHUNK_SIZE):
+    return json_stream.load_many(_to_iterable(response, chunk_size), persistent=persistent, tokenizer=tokenizer)
+
+
 def visit(response, visitor, tokenizer=default_tokenizer, chunk_size=CONTENT_CHUNK_SIZE):
     return json_stream.visit(_to_iterable(response, chunk_size), visitor, tokenizer=tokenizer)
+
+
+def visit_many(response, visitor, tokenizer=default_tokenizer, chunk_size=CONTENT_CHUNK_SIZE):
+    return json_stream.visit_many(_to_iterable(response, chunk_size), visitor, tokenizer=tokenizer)

{json-stream-2.1.0 → json_stream-2.4.1}/src/json_stream/httpx/tests/test_httpx.py

@@ -4,7 +4,8 @@ from itertools import zip_longest
 from unittest import TestCase
 from unittest.mock import Mock
 
-from json_stream.httpx import load, visit
+from json_stream import to_standard_types
+from json_stream.httpx import load, visit, load_many, visit_many
 
 
 class TestLoad(TestCase):
@@ -16,13 +17,14 @@ class TestLoad(TestCase):
         args = [iter(iterable)] * n
         return zip_longest(*args, fillvalue="")
 
-    def _create_mock_response(self):
+    def _create_mock_response(self, data=None):
         # httpx iter_bytes returns an iterable of bytes
         response = Mock()
-        data = json.dumps({
-            "a": "a" * io.DEFAULT_BUFFER_SIZE,
-            "b": "b",
-        })
+        if data is None:
+            data = json.dumps({
+                "a": "a" * io.DEFAULT_BUFFER_SIZE,
+                "b": "b",
+            })
         content = ("".join(chunk).encode() for chunk in self.grouper(data, 1024))
         response.iter_bytes.return_value = content
         return response
@@ -69,3 +71,37 @@ class TestLoad(TestCase):
             ('a' * io.DEFAULT_BUFFER_SIZE, ('a',)),
             ('b', ('b',)),
         ], visited)
+
+    def test_load_many(self):
+        expected = [{
+            "a": "a" * io.DEFAULT_BUFFER_SIZE,
+        }, {
+            "b": "b" * io.DEFAULT_BUFFER_SIZE,
+        }]
+        data = "".join(json.dumps(i) for i in expected)
+        response = self._create_mock_response(data=data)
+        count = 0
+        for item, exp in zip(load_many(response), expected):
+            self.assertEqual(exp, to_standard_types(item))
+            count += 1
+        self.assertEqual(count, len(expected))
+
+    def test_visit_many(self):
+        items = [{
+            "a": "a" * io.DEFAULT_BUFFER_SIZE,
+        }, {
+            "b": "b" * io.DEFAULT_BUFFER_SIZE,
+        }]
+        data = "".join(json.dumps(e) for e in items)
+        response = self._create_mock_response(data=data)
+        current_visit = []
+        visits = []
+
+        def visitor(item, path):
+            current_visit.append((item, path))
+        for _ in visit_many(response, visitor):
+            visits.append(current_visit)
+            current_visit = []
+        self.assertEqual(len(visits), len(items))
+        self.assertEqual(visits[0], [('a' * io.DEFAULT_BUFFER_SIZE, ('a',))])
+        self.assertEqual(visits[1], [('b' * io.DEFAULT_BUFFER_SIZE, ('b',))])

json_stream-2.4.1/src/json_stream/iterators.py

@@ -0,0 +1,33 @@
+import io
+
+
+class IterableStream(io.RawIOBase):
+    def __init__(self, iterable):
+        self.iterator = iter(iterable)
+        self.remainder = None  # type: bytes | None
+
+    def _normalize_chunk(self, chunk):
+        # Ensure chunk is bytes for writing into a binary buffer
+        if isinstance(chunk, str):
+            return chunk.encode()
+        return chunk
+
+    def readinto(self, buffer):
+        try:
+            # Wrap `buffer: WriteableBuffer` in memoryview to ensure len() and slicing
+            mv = memoryview(buffer)
+            chunk = self.remainder or self._normalize_chunk(next(self.iterator))
+            length = min(len(mv), len(chunk))
+            mv[:length], self.remainder = chunk[:length], chunk[length:]
+            return length
+        except StopIteration:
+            return 0    # indicate EOF
+
+    def readable(self):
+        return True
+
+
+def ensure_file(fp_or_iterable):
+    if hasattr(fp_or_iterable, 'read'):
+        return fp_or_iterable
+    return IterableStream(fp_or_iterable)  # will raise TypeError if not iterable

json_stream-2.4.1/src/json_stream/loader.py

@@ -0,0 +1,19 @@
+from json_stream.base import StreamingJSONBase, TokenType
+from json_stream.iterators import ensure_file
+from json_stream.select_tokenizer import default_tokenizer
+
+
+def load(fp_or_iterable, persistent=False, tokenizer=default_tokenizer):
+    return next(load_many(fp_or_iterable, persistent, tokenizer))
+
+
+def load_many(fp_or_iterable, persistent=False, tokenizer=default_tokenizer):
+    fp = ensure_file(fp_or_iterable)
+    token_stream = tokenizer(fp)
+    for token_type, token in token_stream:
+        if token_type == TokenType.OPERATOR:
+            data = StreamingJSONBase.factory(token, token_stream, persistent)
+            yield data
+            data.read_all()
+        else:
+            yield token