omni-json-db 2.7.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- omni_json_db-2.7.0/LICENSE +21 -0
- omni_json_db-2.7.0/PKG-INFO +665 -0
- omni_json_db-2.7.0/README.rst +628 -0
- omni_json_db-2.7.0/omni_json_db/__init__.py +32 -0
- omni_json_db-2.7.0/omni_json_db/jdb.py +4589 -0
- omni_json_db-2.7.0/omni_json_db/jdb_file.py +617 -0
- omni_json_db-2.7.0/omni_json_db/jdb_io.py +2623 -0
- omni_json_db-2.7.0/omni_json_db/jdb_lite.py +3495 -0
- omni_json_db-2.7.0/omni_json_db/jdb_net.py +913 -0
- omni_json_db-2.7.0/omni_json_db/utils.py +328 -0
- omni_json_db-2.7.0/omni_json_db.egg-info/PKG-INFO +665 -0
- omni_json_db-2.7.0/omni_json_db.egg-info/SOURCES.txt +16 -0
- omni_json_db-2.7.0/omni_json_db.egg-info/dependency_links.txt +1 -0
- omni_json_db-2.7.0/omni_json_db.egg-info/requires.txt +11 -0
- omni_json_db-2.7.0/omni_json_db.egg-info/top_level.txt +1 -0
- omni_json_db-2.7.0/pyproject.toml +63 -0
- omni_json_db-2.7.0/setup.cfg +4 -0
- omni_json_db-2.7.0/tests/test.py +6521 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Lukatrum
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,665 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: omni_json_db
|
|
3
|
+
Version: 2.7.0
|
|
4
|
+
Summary: A zero-config, serverless high-performance JSON database with compression. No schema, no setup, just data.
|
|
5
|
+
Author-email: Lukatrum <lukatrum@gmail.com>
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/lukatrum/omni-json-db
|
|
8
|
+
Project-URL: Bug Tracker, https://github.com/lukatrum/omni-json-db/issues
|
|
9
|
+
Keywords: database,JSON,msgpack,marshal,pickle,nosql,memory,network,compression,big data,storage
|
|
10
|
+
Classifier: Programming Language :: Python :: 3
|
|
11
|
+
Classifier: Programming Language :: Python :: 3.7
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.8
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
19
|
+
Classifier: Operating System :: OS Independent
|
|
20
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
21
|
+
Classifier: Intended Audience :: Developers
|
|
22
|
+
Classifier: Topic :: Database
|
|
23
|
+
Classifier: Topic :: Utilities
|
|
24
|
+
Requires-Python: >=3.7
|
|
25
|
+
Description-Content-Type: text/x-rst
|
|
26
|
+
License-File: LICENSE
|
|
27
|
+
Requires-Dist: bitarray>=3.8.0
|
|
28
|
+
Requires-Dist: brotli>=1.2.0
|
|
29
|
+
Requires-Dist: msgpack>=1.0.5
|
|
30
|
+
Requires-Dist: zstandard>=0.21.0
|
|
31
|
+
Requires-Dist: lz4>=4.3.2
|
|
32
|
+
Requires-Dist: orjson>=3.9.7
|
|
33
|
+
Requires-Dist: ormsgpack>=1.2.6
|
|
34
|
+
Requires-Dist: BTrees>=5.2
|
|
35
|
+
Requires-Dist: portalocker>=2.7.0; os_name == "nt"
|
|
36
|
+
Dynamic: license-file
|
|
37
|
+
|
|
38
|
+
👉 Quick Links
|
|
39
|
+
**************
|
|
40
|
+
|
|
41
|
+
- `📌 Supported Python Versions`_
|
|
42
|
+
- `🛠️ Quick Start`_
|
|
43
|
+
- `📝 Specifications`_
|
|
44
|
+
- `📊 Benchmarking`_
|
|
45
|
+
- `👥 Contributing`_
|
|
46
|
+
|
|
47
|
+
✨ Introduction
|
|
48
|
+
***************
|
|
49
|
+
``omni-json-db`` is a high-performance, embedded database engine designed for Python developers who need the speed of a Key-Value store with the querying power of a document database. Built for extreme throughput and thread-safety, ``omni-json-db`` leverages modern serialization (``json``, ``msgpack``, ``marshal``, ``pickle``) and compression to provide a storage layer that is often significantly faster than SQLite for JSON-heavy workloads. Whether you are building a local cache, a log aggregator, or a distributed microservice, ``omni-json-db`` provides the tools to handle data at scale with "Zero-Config" simplicity.
|
|
50
|
+
|
|
51
|
+
* **Schema-LESS**: Store complex, nested data without pre-defining tables.
|
|
52
|
+
|
|
53
|
+
* **Server-LESS**: Direct disk access without the overhead of a database server.
|
|
54
|
+
|
|
55
|
+
* **SQL-LESS**: Use native Python syntax, Regex, and Lambdas for data manipulation.
|
|
56
|
+
|
|
57
|
+
🚀 Features
|
|
58
|
+
***********
|
|
59
|
+
|
|
60
|
+
* **Extreme Performance**: Leverages ``orjson`` and ``ormsgpack`` for serialization. [refer to `Supported Data Formats`_]
|
|
61
|
+
|
|
62
|
+
* **Concurrency Control**: Optimized for Many-Read / Single-Write environments using a robust file-locking and Lock mechanism.
|
|
63
|
+
|
|
64
|
+
* **Advanced Compression**: Supports LZ4 (speed-focused), Zstandard (balanced), and Brotli (size-focused) to minimize storage footprint. [refer to `Supported Zip Formats`_]
|
|
65
|
+
|
|
66
|
+
* **Powerful Querying**: Search using Regular Expressions (RE), Lambda filters, or modification timestamps (Time-Travel query).
|
|
67
|
+
|
|
68
|
+
* **Memory Caching**: Adjustable cache_limit to balance RAM usage and I/O speed.
|
|
69
|
+
|
|
70
|
+
* **Network Mode** (``JNetFiles``): Transform a local ``omni-json-db`` instance into a networked service with a single command using run_files_server. [refer to `Network Mode`_]
|
|
71
|
+
|
|
72
|
+
* **In-Memory Mode** (``JMemFiles``): Run the entire database in RAM for extreme performance (ideal for real-time caches or volatile session storage). [refer to `In-memory Mode`_]
|
|
73
|
+
|
|
74
|
+
* **Revertable**: Unlike traditional NoSQL stores, ``omni-json-db`` tracks internal states allowing you to unwrite (rollback a modification) or undelete a record. This provides a safety net similar to a manual "Undo" or a lightweight ACID rollback. [ref to `Rollback data`_]
|
|
75
|
+
|
|
76
|
+
* **Native CSV Support**: Built-in hooks for DictReader and DictWriter allow you to import massive datasets from CSV files or export your ``omni-json-db`` collections for analysis in Excel or Pandas.
|
|
77
|
+
|
|
78
|
+
* **Date-Based Lookups**: Every record is timestamped, enabling queries like "Give me all users modified last Tuesday." [refer to `Date Lookups`_]
|
|
79
|
+
|
|
80
|
+
📌 Supported Python Versions
|
|
81
|
+
****************************
|
|
82
|
+
|
|
83
|
+
``omni-json-db`` has been tested with Python 3.7 - 3.14.
|
|
84
|
+
|
|
85
|
+
🛠️ Quick Start
|
|
86
|
+
**************
|
|
87
|
+
|
|
88
|
+
Installation
|
|
89
|
+
------------
|
|
90
|
+
|
|
91
|
+
.. code-block:: bash
|
|
92
|
+
|
|
93
|
+
pip install omni-json-db
|
|
94
|
+
|
|
95
|
+
Basic usage
|
|
96
|
+
-----------
|
|
97
|
+
|
|
98
|
+
.. code-block:: python
|
|
99
|
+
|
|
100
|
+
from omni_json_db import JDb
|
|
101
|
+
# Initialize the database from file
|
|
102
|
+
# Key-Value is Json+Json without compression
|
|
103
|
+
jdb = JDb("example.jdb")
|
|
104
|
+
|
|
105
|
+
# Store data
|
|
106
|
+
jdb["user:001"] = {"name" : "Ryan", "role": "Developer"}
|
|
107
|
+
|
|
108
|
+
# Retrieve data
|
|
109
|
+
user = jdb["user:001"]
|
|
110
|
+
print(user["name"]) # Output: Ryan
|
|
111
|
+
|
|
112
|
+
All standard ``dict`` methods work: ``keys()``, ``values()``, ``items()``, ``pop()``, ``setdefault()``, ``update()``.
|
|
113
|
+
|
|
114
|
+
In-memory Mode
|
|
115
|
+
--------------
|
|
116
|
+
|
|
117
|
+
.. code-block:: python
|
|
118
|
+
|
|
119
|
+
from omni_json_db import JDb
|
|
120
|
+
# Initialize the database in memory
|
|
121
|
+
# Key-Value is Json+Msgpack with Gzip compression
|
|
122
|
+
jdb = JDb(data_type="J+S", zip_type="gz")
|
|
123
|
+
|
|
124
|
+
# Store data
|
|
125
|
+
jdb += {"user:001" : {"name" : "Joe", "role": "Senior Developer"}}
|
|
126
|
+
|
|
127
|
+
# Retrieve data
|
|
128
|
+
user = jdb["user:001"]
|
|
129
|
+
print(user["name"]) # Output: Joe
|
|
130
|
+
|
|
131
|
+
# create 2nd JDb with same memory
|
|
132
|
+
jdb2 = JDb(jdb)
|
|
133
|
+
|
|
134
|
+
# Store data
|
|
135
|
+
jdb2["user:002"] ={"name" : "Kathy", "role": "CEO"}
|
|
136
|
+
|
|
137
|
+
assert jdb == jdb2
|
|
138
|
+
assert len(jdb) == 2
|
|
139
|
+
print(jdb["user:002"]["name"]) # Output: Kathy
|
|
140
|
+
print(set(jdb)) # Output: {'user:001', 'user:002'}
|
|
141
|
+
|
|
142
|
+
Rollback data
|
|
143
|
+
-------------
|
|
144
|
+
|
|
145
|
+
.. code-block:: python
|
|
146
|
+
|
|
147
|
+
from omni_json_db import JDb
|
|
148
|
+
# Initialize the database from file
|
|
149
|
+
# Key-Value is Json+Pickle with zstandard compression
|
|
150
|
+
jdb = JDb("fruit.jdb", data_type="J+P(zs)")
|
|
151
|
+
|
|
152
|
+
jdb["apple"] = "red"
|
|
153
|
+
jdb["apple"] = "blue" # modify
|
|
154
|
+
jdb.revert("apple") # unmodify
|
|
155
|
+
assert jdb["apple"] == 'red'
|
|
156
|
+
|
|
157
|
+
del jdb["apple"]
|
|
158
|
+
assert "apple" not in jdb
|
|
159
|
+
|
|
160
|
+
jdb.revert("apple") # unremove
|
|
161
|
+
assert jdb["apple"] == "red"
|
|
162
|
+
|
|
163
|
+
Query data
|
|
164
|
+
----------
|
|
165
|
+
|
|
166
|
+
.. code-block:: python
|
|
167
|
+
|
|
168
|
+
from omni_json_db import JDb
|
|
169
|
+
# Initialize the database in memory
|
|
170
|
+
# Key-Value is Json+Marshal with no compression
|
|
171
|
+
jdb = JDb(data_type="J+M")
|
|
172
|
+
|
|
173
|
+
# insert value without key
|
|
174
|
+
jdb += [{'name': 'John', 'age': 22}, {'name': 'John', 'age': 37}, \
|
|
175
|
+
{'name': 'Bob', 'age': 42}, {'name': 'Megan', 'age': 27}]
|
|
176
|
+
|
|
177
|
+
print(jdb[:]) # print all records from jdb
|
|
178
|
+
|
|
179
|
+
matches = jdb.find(FUNC=lambda k,v: v.get('name', '') == 'John')
|
|
180
|
+
print(matches) # Output : {'0': {'name': 'John', 'age': 22}, '1': {'name': 'John', 'age': 37}}
|
|
181
|
+
|
|
182
|
+
matches = jdb.find(RE='John|Bob')
|
|
183
|
+
print(matches) # {'0': {'name': 'John', 'age': 22}, '1': {'name': 'John', 'age': 37}, '2': {'name': 'Bob', 'age': 42}}
|
|
184
|
+
|
|
185
|
+
Operator
|
|
186
|
+
--------
|
|
187
|
+
|
|
188
|
+
.. code-block:: python
|
|
189
|
+
|
|
190
|
+
from omni_json_db import JDb
|
|
191
|
+
# Initialize the database in memory
|
|
192
|
+
# Key+Value is Msgpack+Msgpack with lz4 compression
|
|
193
|
+
jdb = JDb(data_type="S+S(lz)")
|
|
194
|
+
|
|
195
|
+
# [1] KEY+VAL operators
|
|
196
|
+
# <jdb += data> == jdb.update(data)
|
|
197
|
+
data = {f'key{v}':v for v in range(100)}
|
|
198
|
+
jdb += data
|
|
199
|
+
assert len(jdb) == 100
|
|
200
|
+
|
|
201
|
+
# <jdb == data>
|
|
202
|
+
assert jdb == data
|
|
203
|
+
|
|
204
|
+
# <jdb |= ..> == jdb.insert(..)
|
|
205
|
+
jdb |= {f'key{v}':v+1 for v in range(102)}
|
|
206
|
+
assert len(jdb) == 102
|
|
207
|
+
assert jdb['key100'] == 101
|
|
208
|
+
assert jdb[-2.:] == {'key100':101, 'key101':102} # get last two modified records
|
|
209
|
+
assert jdb[(f'key{v}' for v in range(100))] == data # same as jdb[data] == data
|
|
210
|
+
|
|
211
|
+
# <jdb -= ..> == jdb.remove(..)
|
|
212
|
+
jdb -= ['key100', 'key101', 'key102', 'key103']
|
|
213
|
+
assert len(jdb) == 100
|
|
214
|
+
assert jdb == data
|
|
215
|
+
|
|
216
|
+
# <jdb &= ..> == jdb.replace(..)
|
|
217
|
+
jdb &= {f'key{v}':v+1 for v in range(200)}
|
|
218
|
+
assert len(jdb) == 100
|
|
219
|
+
assert jdb == {f'key{v}':v+1 for v in range(100)}
|
|
220
|
+
|
|
221
|
+
# <jdb ^= ..> == jdb.unmodify(..)
|
|
222
|
+
jdb ^= {f'key{v}' for v in range(100)} # same as jdb ^= data
|
|
223
|
+
assert len(jdb) == 100
|
|
224
|
+
assert jdb == data
|
|
225
|
+
|
|
226
|
+
# <jdb[:] = ..> == jdb.update(..)
|
|
227
|
+
jdb[:] = 0 # set all records to zero
|
|
228
|
+
assert len(jdb) == 100
|
|
229
|
+
assert jdb == {f'key{v}':0 for v in range(100)}
|
|
230
|
+
assert jdb.find(NE=0) == {}
|
|
231
|
+
|
|
232
|
+
# remove all records
|
|
233
|
+
jdb -= jdb # same as del jdb[:]
|
|
234
|
+
assert len(jdb) == 0
|
|
235
|
+
|
|
236
|
+
# <jdb ^= ..> == jdb.unremove(..)
|
|
237
|
+
jdb ^= {f'key{v}' for v in range(100)} # same as jdb ^= data
|
|
238
|
+
assert len(jdb) == 100
|
|
239
|
+
assert all(val == 0 for key,val in jdb.items())
|
|
240
|
+
|
|
241
|
+
# lambda VALUE operation
|
|
242
|
+
jdb[:] = lambda key,val: int(key.replace('key', '')) + val
|
|
243
|
+
assert jdb == data
|
|
244
|
+
|
|
245
|
+
# <del jdb[..]> == jdb.remove_fast(..)
|
|
246
|
+
del jdb[data] # same as del jdb[:]
|
|
247
|
+
assert len(jdb) == 0
|
|
248
|
+
|
|
249
|
+
# unremove all data
|
|
250
|
+
jdb ^= data
|
|
251
|
+
assert jdb == data
|
|
252
|
+
|
|
253
|
+
# <jdb[..]> == jdb.get_n(..) or jdb.get_all()
|
|
254
|
+
matches = jdb[('key2', 'key22', 'key44', 'key111')]
|
|
255
|
+
assert matches == {'key2':2, 'key22':22, 'key44':44}
|
|
256
|
+
|
|
257
|
+
# lambda KEY operation
|
|
258
|
+
matches = jdb[lambda key:key.endswith('1')]
|
|
259
|
+
assert set(matches) == {'key1', 'key11', 'key21', 'key31', 'key41', 'key51', 'key61', 'key71', 'key81', 'key91'}
|
|
260
|
+
|
|
261
|
+
# set all matched records to -1
|
|
262
|
+
jdb[matches] = -1
|
|
263
|
+
matches_2 = jdb[lambda key,val: val == -1]
|
|
264
|
+
assert set(matches) == set(matches_2)
|
|
265
|
+
assert matches_2 == jdb.find(EQ=-1)
|
|
266
|
+
assert matches_2 == jdb.find(FUNC=lambda val: val == -1)
|
|
267
|
+
|
|
268
|
+
# RE search
|
|
269
|
+
matches_3 = jdb[::r'1$']
|
|
270
|
+
assert matches_2 == matches_3
|
|
271
|
+
|
|
272
|
+
# unmodify
|
|
273
|
+
jdb ^= matches
|
|
274
|
+
assert jdb == data
|
|
275
|
+
|
|
276
|
+
# [2] KEY operators
|
|
277
|
+
# <jdb & {..}> == jdb.intersection(..)
|
|
278
|
+
matches = jdb & {f'key{v}' for v in range(98, 120)}
|
|
279
|
+
assert matches == {'key98', 'key99'}
|
|
280
|
+
|
|
281
|
+
# <{..} & jdb> == {..}.intersection(jdb)
|
|
282
|
+
matches_2 = {f'key{v}' for v in range(98, 120)} & jdb
|
|
283
|
+
assert matches == matches_2
|
|
284
|
+
|
|
285
|
+
# <jdb | {..}> == jdb.union(..)
|
|
286
|
+
matches = jdb | {f'key{v}' for v in range(10, 120)}
|
|
287
|
+
assert len(matches) == 120
|
|
288
|
+
assert matches == {f'key{v}' for v in range(0, 120)}
|
|
289
|
+
|
|
290
|
+
# <{..} | jdb> == {..}.union(jdb)
|
|
291
|
+
matches_2 = {f'key{v}' for v in range(10, 120)} | jdb
|
|
292
|
+
assert matches == matches_2
|
|
293
|
+
|
|
294
|
+
# <jdb + {..}> == jdb.union(..)
|
|
295
|
+
matches = jdb + {f'key{v}' for v in range(10, 120)}
|
|
296
|
+
assert matches == matches_2
|
|
297
|
+
|
|
298
|
+
# <{..} + jdb> == {..}.union(jdb)
|
|
299
|
+
matches_2 = {f'key{v}' for v in range(10, 120)} + jdb
|
|
300
|
+
assert matches == matches_2
|
|
301
|
+
|
|
302
|
+
# <jdb - {..}> == jdb.difference(..)
|
|
303
|
+
matches = jdb - {f'key{v}' for v in range(0, 98)}
|
|
304
|
+
assert matches == {'key98', 'key99'}
|
|
305
|
+
|
|
306
|
+
# <{..} - jdb> == {..}.difference(jdb)
|
|
307
|
+
matches = {f'key{v}' for v in range(2, 102)} - jdb
|
|
308
|
+
assert matches == {'key100', 'key101'}
|
|
309
|
+
|
|
310
|
+
# <jdb ^ {..}> == jdb.non_intersection(..)
|
|
311
|
+
matches = jdb ^ {f'key{v}' for v in range(1, 101)}
|
|
312
|
+
assert matches == {'key0', 'key100'}
|
|
313
|
+
|
|
314
|
+
# <{..} ^ jdb> == {..}.non_intersection(jdb)
|
|
315
|
+
matches_2 = {f'key{v}' for v in range(1, 101)} ^ jdb
|
|
316
|
+
assert matches == matches_2
|
|
317
|
+
|
|
318
|
+
# <.. in jdb> == jdb.has_all(..)
|
|
319
|
+
assert 'key10' in jdb
|
|
320
|
+
assert {'key10', 'key90'} in jdb
|
|
321
|
+
assert {'key10', 'key90', 'key110', 'key190'} not in jdb
|
|
322
|
+
assert jdb.has('key10')
|
|
323
|
+
assert jdb.has_all('key10')
|
|
324
|
+
assert jdb.has_any('key10')
|
|
325
|
+
assert jdb.has_all({'key10', 'key90'})
|
|
326
|
+
assert jdb.has_any({'key10', 'key90', 'key110', 'key190'})
|
|
327
|
+
assert jdb.is_disjoint({'key110', 'key190'})
|
|
328
|
+
|
|
329
|
+
Date Lookups
|
|
330
|
+
------------
|
|
331
|
+
|
|
332
|
+
.. code-block:: python
|
|
333
|
+
|
|
334
|
+
from omni_json_db import JDb
|
|
335
|
+
import datetime as dt
|
|
336
|
+
|
|
337
|
+
# Initialize the database in memory
|
|
338
|
+
# Key+Value is Json+Msgpack with Brotli compression
|
|
339
|
+
# using BTree as Key Table for better memory usage
|
|
340
|
+
jdb = JDb(data_type="J+S(br)", key_limit="bt")
|
|
341
|
+
|
|
342
|
+
# insert data
|
|
343
|
+
fruits = {'apple':'red', 'banana':'yellow', 'mango':'yellow', 'lemon':'yellow', 'tomato':'red'}
|
|
344
|
+
jdb += fruits
|
|
345
|
+
|
|
346
|
+
# datetime for create date, date for modify date
|
|
347
|
+
now = dt.datetime.now()
|
|
348
|
+
today = now.date()
|
|
349
|
+
|
|
350
|
+
# find create date: date == now
|
|
351
|
+
matches = jdb[now]
|
|
352
|
+
assert matches == fruits
|
|
353
|
+
|
|
354
|
+
# find create date: date >= now
|
|
355
|
+
matches = jdb[now:]
|
|
356
|
+
assert matches == fruits
|
|
357
|
+
|
|
358
|
+
# find create date: date < now
|
|
359
|
+
matches = jdb[:now]
|
|
360
|
+
assert len(matches) == 0
|
|
361
|
+
|
|
362
|
+
# find create date: now <= date <= now+1
|
|
363
|
+
next_date = now + dt.timedelta(days=1)
|
|
364
|
+
matches = jdb[now:next_date]
|
|
365
|
+
assert matches == fruits
|
|
366
|
+
|
|
367
|
+
prev_date = now - dt.timedelta(days=1)
|
|
368
|
+
prev_week = now - dt.timedelta(days=7)
|
|
369
|
+
|
|
370
|
+
# change key create date
|
|
371
|
+
jdb.keys['apple', 'tomato'] = prev_date
|
|
372
|
+
jdb.keys['mango'] = prev_week
|
|
373
|
+
assert jdb[prev_date] == {'apple':'red', 'tomato':'red'}
|
|
374
|
+
assert jdb[prev_week] == {'mango':'yellow'}
|
|
375
|
+
|
|
376
|
+
# find create date: date == now
|
|
377
|
+
matches = jdb[now]
|
|
378
|
+
assert set(matches) == {'banana', 'lemon'}
|
|
379
|
+
|
|
380
|
+
# find create date: date < now
|
|
381
|
+
matches = jdb[:now]
|
|
382
|
+
assert set(matches) == {'apple', 'mango', 'tomato'}
|
|
383
|
+
|
|
384
|
+
# find modify date: date == today
|
|
385
|
+
matches = jdb[today]
|
|
386
|
+
assert matches == fruits
|
|
387
|
+
|
|
388
|
+
# change key modify date + create date
|
|
389
|
+
new_modify_date = prev_date.date()
|
|
390
|
+
new_create_date = prev_week.date()
|
|
391
|
+
assert new_modify_date >= new_create_date
|
|
392
|
+
jdb.keys['lemon'] = f'{new_modify_date} {new_create_date}'
|
|
393
|
+
|
|
394
|
+
# find modify date: date == today
|
|
395
|
+
matches = jdb[today]
|
|
396
|
+
assert set(matches) == {'apple', 'banana', 'mango', 'tomato'}
|
|
397
|
+
|
|
398
|
+
# find modify date: date == prev_date
|
|
399
|
+
matches = jdb[prev_date.date()]
|
|
400
|
+
assert set(matches) == {'lemon'}
|
|
401
|
+
|
|
402
|
+
# change all keys create date
|
|
403
|
+
jdb.keys[:] = today
|
|
404
|
+
assert jdb[today] == fruits
|
|
405
|
+
|
|
406
|
+
Advanced Usage
|
|
407
|
+
--------------
|
|
408
|
+
|
|
409
|
+
.. code-block:: python
|
|
410
|
+
|
|
411
|
+
from omni_json_db import JDb
|
|
412
|
+
|
|
413
|
+
jdb = JDb()
|
|
414
|
+
|
|
415
|
+
fruits = {'apple':'red', 'banana':'yellow', 'mango':'yellow', 'lemon':'yellow', 'tomato':'red'}
|
|
416
|
+
|
|
417
|
+
# insert records
|
|
418
|
+
with jdb.open() as fp:
|
|
419
|
+
for fruit,color in fruits.items():
|
|
420
|
+
jdb._write(fp, fruit, color)
|
|
421
|
+
|
|
422
|
+
assert jdb == fruits
|
|
423
|
+
|
|
424
|
+
# modify records
|
|
425
|
+
with jdb.open() as fp:
|
|
426
|
+
for fruit in fruits:
|
|
427
|
+
color = jdb._read(fp, fruit)
|
|
428
|
+
jdb._write(fp, fruit, color.upper())
|
|
429
|
+
|
|
430
|
+
assert jdb != fruits
|
|
431
|
+
assert set(jdb) == set(fruits)
|
|
432
|
+
|
|
433
|
+
# unmodify records
|
|
434
|
+
with jdb.open() as fp:
|
|
435
|
+
for fruit in fruits:
|
|
436
|
+
jdb._unwrite(fp, fruit)
|
|
437
|
+
|
|
438
|
+
assert jdb == fruits
|
|
439
|
+
|
|
440
|
+
# remove records
|
|
441
|
+
with jdb.open() as fp:
|
|
442
|
+
for fruit in fruits:
|
|
443
|
+
jdb._delete(fp, fruit)
|
|
444
|
+
|
|
445
|
+
assert len(jdb) == 0
|
|
446
|
+
|
|
447
|
+
# unremove records
|
|
448
|
+
with jdb.open() as fp:
|
|
449
|
+
for fruit in fruits:
|
|
450
|
+
jdb._undelete(fp, fruit)
|
|
451
|
+
|
|
452
|
+
assert jdb == fruits
|
|
453
|
+
|
|
454
|
+
#---------------------------------------
|
|
455
|
+
with jdb.open() as fp:
|
|
456
|
+
key_table = jdb.key_table
|
|
457
|
+
|
|
458
|
+
# replace
|
|
459
|
+
for fruit in key_table:
|
|
460
|
+
color = jdb._read(fp, fruit)
|
|
461
|
+
jdb._write(fp, fruit, color.upper())
|
|
462
|
+
|
|
463
|
+
# unmodify
|
|
464
|
+
for fruit in key_table:
|
|
465
|
+
jdb._unwrite(fp, fruit)
|
|
466
|
+
|
|
467
|
+
# remove
|
|
468
|
+
for fruit in fruits:
|
|
469
|
+
jdb._delete(fp, fruit)
|
|
470
|
+
|
|
471
|
+
# unremove
|
|
472
|
+
for fruit in fruits:
|
|
473
|
+
jdb._undelete(fp, fruit)
|
|
474
|
+
|
|
475
|
+
assert jdb == fruits
|
|
476
|
+
|
|
477
|
+
#---------------------------------------
|
|
478
|
+
# replace all
|
|
479
|
+
jdb[:] = lambda k,v: v.upper()
|
|
480
|
+
|
|
481
|
+
# unmodify all
|
|
482
|
+
jdb ^= jdb
|
|
483
|
+
|
|
484
|
+
# remove all
|
|
485
|
+
jdb -= jdb
|
|
486
|
+
|
|
487
|
+
# unremove all
|
|
488
|
+
jdb ^= fruits
|
|
489
|
+
|
|
490
|
+
assert jdb == fruits
|
|
491
|
+
|
|
492
|
+
Network Mode
|
|
493
|
+
-------------
|
|
494
|
+
|
|
495
|
+
**Server side:**
|
|
496
|
+
|
|
497
|
+
.. code-block:: python
|
|
498
|
+
|
|
499
|
+
>> from omni_json_db import run_files_server
|
|
500
|
+
>> run_files_server(host='0.0.0.0', port=59698, files='net_storage.jdb')
|
|
501
|
+
|
|
502
|
+
**Client side:**
|
|
503
|
+
|
|
504
|
+
.. code-block:: python
|
|
505
|
+
|
|
506
|
+
>> from omni_json_db import JDb, JNetFiles
|
|
507
|
+
>> jdb = JDb(JNetFiles(('0.0.0.0', 59898)))
|
|
508
|
+
|
|
509
|
+
📝 Specifications
|
|
510
|
+
*****************
|
|
511
|
+
|
|
512
|
+
Supported Data Formats
|
|
513
|
+
----------------------
|
|
514
|
+
|
|
515
|
+
Configure ``data_type`` during initialization:
|
|
516
|
+
|
|
517
|
+
* ``J+J``: JSON Key + JSON Value (default)
|
|
518
|
+
* ``J+S``: JSON Key + MsgPack Value
|
|
519
|
+
* ``J+M``: JSON Key + Marshal Value
|
|
520
|
+
* ``J+P``: JSON Key + Pickle Value
|
|
521
|
+
* ``S+J``: MsgPack Key + JSON Value
|
|
522
|
+
* ``S+S``: MsgPack Key + MsgPack Value
|
|
523
|
+
* ``S+M``: MsgPack Key + Marshal Value
|
|
524
|
+
* ``S+P``: MsgPack Key + Pickle Value
|
|
525
|
+
|
|
526
|
+
**Data size = 70,840,580 (MB = 1,000,000B, no zip)**
|
|
527
|
+
|
|
528
|
+
+-------------------+------------+-------+----------+-----------+----------------+---------------+
|
|
529
|
+
| ``data_type`` | size | ratio | read | write | GOODs | BADs |
|
|
530
|
+
+===================+============+=======+==========+===========+================+===============+
|
|
531
|
+
| ``J+J`` or ``S+J``| 70,840,580 | 1.00 | 75.3MB/s | 358.0MB/s |* fastest write |* no set() |
|
|
532
|
+
| | | | | |* faster read |* no tuple() |
|
|
533
|
+
| | | | | |* readable |* weak bytes |
|
|
534
|
+
+-------------------+------------+-------+----------+-----------+----------------+---------------+
|
|
535
|
+
| ``J+S`` or ``S+S``| 47,616,008 | 1.48 | 77.4MB/s | 354.2MB/s |* smallest size |* no tuple() |
|
|
536
|
+
| | | | | |* faster read |* unreadable |
|
|
537
|
+
| | | | | |* faster write | |
|
|
538
|
+
+-------------------+------------+-------+----------+-----------+----------------+---------------+
|
|
539
|
+
| ``J+M`` or ``S+M``| 72,430,958 | 0.97 | 81.4MB/s | 177.1MB/s |* all type [1]_ |* biggest size |
|
|
540
|
+
| | | | | |* fastest read |* unreadable |
|
|
541
|
+
+-------------------+------------+-------+----------+-----------+----------------+---------------+
|
|
542
|
+
| ``J+P`` or ``S+P``| 70,207,207 | 1.01 | 64.9MB/s | 22.8MB/s |* all type [1]_ |* slowest read |
|
|
543
|
+
| | | | | | |* slowest write|
|
|
544
|
+
| | | | | | |* unreadable |
|
|
545
|
+
+-------------------+------------+-------+----------+-----------+----------------+---------------+
|
|
546
|
+
|
|
547
|
+
.. [1] all type = ``str``, ``bytes``, ``bool``, ``int``, ``float``, ``list``, ``tuple``, ``set``, ``dict``, ``None``
|
|
548
|
+
|
|
549
|
+
Supported Zip Formats
|
|
550
|
+
---------------------
|
|
551
|
+
|
|
552
|
+
Configure ``zip_type`` during initialization:
|
|
553
|
+
|
|
554
|
+
* ``no``: no compression for Value (default)
|
|
555
|
+
* ``gz``: Gzip (mode=9) compression for Value
|
|
556
|
+
* ``bz``: Bzip2 (mode=9) compression for Value
|
|
557
|
+
* ``xz``: LZMA compression for Value
|
|
558
|
+
* ``zs``: Zstandard (mode=22) compression for Value
|
|
559
|
+
* ``br``: Brotli (mode=6) compression for Value (better than ``gz``)
|
|
560
|
+
* ``z1``: Zstandard (mode=6) compression for Value (better than ``gz``)
|
|
561
|
+
* ``z2``: Zstandard (mode=11) compression for Value
|
|
562
|
+
* ``lz``: LZ4 (mode=0) compression for Value
|
|
563
|
+
|
|
564
|
+
**Data size = 70,840,580 (MB = 1,000,000B)**
|
|
565
|
+
|
|
566
|
+
+------------+------------+-------+----------+-----------+---------------+---------------+
|
|
567
|
+
|``zip_type``| size | ratio | read | write | GOODs | BADs |
|
|
568
|
+
+============+============+=======+==========+===========+===============+===============+
|
|
569
|
+
| ``no`` | 70,840,580 | 1.00 | 75.3MB/s | 358.0MB/s |* fastest speed|* biggest size |
|
|
570
|
+
+------------+------------+-------+----------+-----------+---------------+---------------+
|
|
571
|
+
| ``gz`` | 16,915,844 | 4.18 | 65.5MB/s | 5.1MB/s | |* slower zip |
|
|
572
|
+
+------------+------------+-------+----------+-----------+---------------+---------------+
|
|
573
|
+
| ``bz`` | 11,394,042 | 6.21 | 26.4MB/s | 10.8MB/s |* better ratio |* slowest unzip|
|
|
574
|
+
+------------+------------+-------+----------+-----------+---------------+---------------+
|
|
575
|
+
| ``xz`` | 11,340,548 | 6.24 | 54.9MB/s | 2.3MB/s |* better ratio |* slower zip |
|
|
576
|
+
| | | | | | |* slower unzip |
|
|
577
|
+
+------------+------------+-------+----------+-----------+---------------+---------------+
|
|
578
|
+
| ``zs`` | 11,119,665 | 6.37 | 73.0MB/s | 1.7MB/s |* best ratio |* slowest zip |
|
|
579
|
+
| | | | | |* faster unzip | |
|
|
580
|
+
+------------+------------+-------+----------+-----------+---------------+---------------+
|
|
581
|
+
| ``br`` | 13,700,696 | 5.17 | 65.8MB/s | 25.3MB/s |* better ``gz``| |
|
|
582
|
+
+------------+------------+-------+----------+-----------+---------------+---------------+
|
|
583
|
+
| ``z1`` | 14,738,859 | 4.80 | 73.6MB/s | 70.8MB/s |* faster zip | |
|
|
584
|
+
| | | | | |* faster unzip | |
|
|
585
|
+
+------------+------------+-------+----------+-----------+---------------+---------------+
|
|
586
|
+
| ``z2`` | 13,799,407 | 5.13 | 72.7MB/s | 23.6MB/s |* faster unzip | |
|
|
587
|
+
+------------+------------+-------+----------+-----------+---------------+---------------+
|
|
588
|
+
| ``lz`` | 26,226,039 | 2.70 | 75.6MB/s | 202.4MB/s |* fastest zip |* worst ratio |
|
|
589
|
+
| | | | | |* fastest unzip| |
|
|
590
|
+
+------------+------------+-------+----------+-----------+---------------+---------------+
|
|
591
|
+
|
|
592
|
+
Supported Key Table Formats
|
|
593
|
+
---------------------------
|
|
594
|
+
|
|
595
|
+
Configure ``key_limit`` during initialization:
|
|
596
|
+
|
|
597
|
+
* ``no``: ``dict`` for key_table (default)
|
|
598
|
+
* ``bt``: ``BTree`` for key_table (save 44.3% vs ``dict``)
|
|
599
|
+
* ``l0`` - ``l5``: ``LiteKeyTable`` modes (save 60-75%+ vs ``dict``)
|
|
600
|
+
|
|
601
|
+
**Table size = 3,241,854 keys**
|
|
602
|
+
|
|
603
|
+
+---------------+--------+--------------+------------+--------------+
|
|
604
|
+
| ``key_limit`` | memory | key search | HIT > get()| MISS > get() |
|
|
605
|
+
+===============+========+==============+============+==============+
|
|
606
|
+
| ``no`` | 519MB | 48.59Mo/s | 29.28Mo/s | 18.3Mo/s |
|
|
607
|
+
+---------------+--------+--------------+------------+--------------+
|
|
608
|
+
| ``bt`` | 289MB | 3.46Mo/s | 3.07Mo/s | 8.04Mo/s |
|
|
609
|
+
+---------------+--------+--------------+------------+--------------+
|
|
610
|
+
| ``l3`` | 85MB | 2.01Mo/s | 2.01Mo/s | 1.59Mo/s |
|
|
611
|
+
+---------------+--------+--------------+------------+--------------+
|
|
612
|
+
|
|
613
|
+
📊 Benchmarking
|
|
614
|
+
***************
|
|
615
|
+
|
|
616
|
+
Testing
|
|
617
|
+
-------
|
|
618
|
+
|
|
619
|
+
.. code-block:: python
|
|
620
|
+
|
|
621
|
+
>> from omni_json_db import JDb
|
|
622
|
+
>> size = 1_000_000
|
|
623
|
+
>> jdb = JDb(data_type='J+J')
|
|
624
|
+
>> data = {f'key{k}':k for k in range(size)}
|
|
625
|
+
|
|
626
|
+
>> # Benchmarking operations
|
|
627
|
+
>> jdb += data # insert
|
|
628
|
+
>> jdb[:] # get_all
|
|
629
|
+
>> jdb -= data # remove
|
|
630
|
+
>> jdb ^= data # revert=unremove
|
|
631
|
+
>> jdb[data] = -1 # replace
|
|
632
|
+
>> jdb ^= data # revert=unmodify
|
|
633
|
+
>> print(jdb == data) # Output: True
|
|
634
|
+
|
|
635
|
+
Results
|
|
636
|
+
-------
|
|
637
|
+
|
|
638
|
+
+-------+---------+---------+---------+----------+---------+----------+
|
|
639
|
+
| size | insert | get_all | remove | unremove | replace | unmodify |
|
|
640
|
+
+=======+=========+=========+=========+==========+=========+==========+
|
|
641
|
+
| 1 | 132 μs | 89 μs | 111 μs | 96 μs | 91 μs | 83 μs |
|
|
642
|
+
+-------+---------+---------+---------+----------+---------+----------+
|
|
643
|
+
| 10 | 136 μs | 93 μs | 142 μs | 145 μs | 183 μs | 177 μs |
|
|
644
|
+
+-------+---------+---------+---------+----------+---------+----------+
|
|
645
|
+
| 100 | 442 μs | 319 μs | 594 μs | 680 μs | 876 μs | 976 μs |
|
|
646
|
+
+-------+---------+---------+---------+----------+---------+----------+
|
|
647
|
+
| 1K | 3.37 ms | 2.71 ms | 5.24 ms | 5.9 ms | 7.61 ms | 9.12 ms |
|
|
648
|
+
+-------+---------+---------+---------+----------+---------+----------+
|
|
649
|
+
| 10K | 32.2 ms | 26 ms | 54.3 ms | 55.8 ms | 77.5 ms | 91.1 ms |
|
|
650
|
+
+-------+---------+---------+---------+----------+---------+----------+
|
|
651
|
+
| 100K | 358 ms | 262 ms | 626 ms | 583 ms | 774 ms | 930 ms |
|
|
652
|
+
+-------+---------+---------+---------+----------+---------+----------+
|
|
653
|
+
| 1M | 3.87 s | 2.78 s | 7 s | 6.09 s | 8.15 s | 9.83 s |
|
|
654
|
+
+-------+---------+---------+---------+----------+---------+----------+
|
|
655
|
+
|
|
656
|
+
👥 Contributing
|
|
657
|
+
***************
|
|
658
|
+
|
|
659
|
+
Whether reporting bugs, discussing improvements and new ideas or writing extensions: Contributions to ``omni-json-db`` are welcome! Here's how to get started:
|
|
660
|
+
|
|
661
|
+
1. Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug.
|
|
662
|
+
2. Fork `the repository <https://github.com/lukatrum/omni-json-db/>`_ on Github, create a new branch off the ``master`` branch and start making your changes (known as `GitHub Flow <https://docs.github.com/en/get-started/using-github/github-flow>`_).
|
|
663
|
+
3. Write a test which shows that the bug was fixed or that the feature works as expected.
|
|
664
|
+
4. Send a pull request and bug the maintainer until it gets merged and published ☺
|
|
665
|
+
|