omni-json-db 2.7.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Lukatrum
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,665 @@
1
+ Metadata-Version: 2.4
2
+ Name: omni_json_db
3
+ Version: 2.7.0
4
+ Summary: A zero-config, serverless high-performance JSON database with compression. No schema, no setup, just data.
5
+ Author-email: Lukatrum <lukatrum@gmail.com>
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://github.com/lukatrum/omni-json-db
8
+ Project-URL: Bug Tracker, https://github.com/lukatrum/omni-json-db/issues
9
+ Keywords: database,JSON,msgpack,marshal,pickle,nosql,memory,network,compression,big data,storage
10
+ Classifier: Programming Language :: Python :: 3
11
+ Classifier: Programming Language :: Python :: 3.7
12
+ Classifier: Programming Language :: Python :: 3.8
13
+ Classifier: Programming Language :: Python :: 3.9
14
+ Classifier: Programming Language :: Python :: 3.10
15
+ Classifier: Programming Language :: Python :: 3.11
16
+ Classifier: Programming Language :: Python :: 3.12
17
+ Classifier: Programming Language :: Python :: 3.13
18
+ Classifier: Programming Language :: Python :: 3.14
19
+ Classifier: Operating System :: OS Independent
20
+ Classifier: Development Status :: 5 - Production/Stable
21
+ Classifier: Intended Audience :: Developers
22
+ Classifier: Topic :: Database
23
+ Classifier: Topic :: Utilities
24
+ Requires-Python: >=3.7
25
+ Description-Content-Type: text/x-rst
26
+ License-File: LICENSE
27
+ Requires-Dist: bitarray>=3.8.0
28
+ Requires-Dist: brotli>=1.2.0
29
+ Requires-Dist: msgpack>=1.0.5
30
+ Requires-Dist: zstandard>=0.21.0
31
+ Requires-Dist: lz4>=4.3.2
32
+ Requires-Dist: orjson>=3.9.7
33
+ Requires-Dist: ormsgpack>=1.2.6
34
+ Requires-Dist: BTrees>=5.2
35
+ Requires-Dist: portalocker>=2.7.0; os_name == "nt"
36
+ Dynamic: license-file
37
+
38
+ 👉 Quick Links
39
+ **************
40
+
41
+ - `📌 Supported Python Versions`_
42
+ - `🛠️ Quick Start`_
43
+ - `📝 Specifications`_
44
+ - `📊 Benchmarking`_
45
+ - `👥 Contributing`_
46
+
47
+ ✨ Introduction
48
+ ***************
49
+ ``omni-json-db`` is a high-performance, embedded database engine designed for Python developers who need the speed of a Key-Value store with the querying power of a document database. Built for extreme throughput and thread-safety, ``omni-json-db`` leverages modern serialization (``json``, ``msgpack``, ``marshal``, ``pickle``) and compression to provide a storage layer that is often significantly faster than SQLite for JSON-heavy workloads. Whether you are building a local cache, a log aggregator, or a distributed microservice, ``omni-json-db`` provides the tools to handle data at scale with "Zero-Config" simplicity.
50
+
51
+ * **Schema-LESS**: Store complex, nested data without pre-defining tables.
52
+
53
+ * **Server-LESS**: Direct disk access without the overhead of a database server.
54
+
55
+ * **SQL-LESS**: Use native Python syntax, Regex, and Lambdas for data manipulation.
56
+
57
+ 🚀 Features
58
+ ***********
59
+
60
+ * **Extreme Performance**: Leverages ``orjson`` and ``ormsgpack`` for serialization. [refer to `Supported Data Formats`_]
61
+
62
+ * **Concurrency Control**: Optimized for Many-Read / Single-Write environments using a robust file-locking and Lock mechanism.
63
+
64
+ * **Advanced Compression**: Supports LZ4 (speed-focused), Zstandard (balanced), and Brotli (size-focused) to minimize storage footprint. [refer to `Supported Zip Formats`_]
65
+
66
+ * **Powerful Querying**: Search using Regular Expressions (RE), Lambda filters, or modification timestamps (Time-Travel query).
67
+
68
+ * **Memory Caching**: Adjustable cache_limit to balance RAM usage and I/O speed.
69
+
70
+ * **Network Mode** (``JNetFiles``): Transform a local ``omni-json-db`` instance into a networked service with a single command using run_files_server. [refer to `Network Mode`_]
71
+
72
+ * **In-Memory Mode** (``JMemFiles``): Run the entire database in RAM for extreme performance (ideal for real-time caches or volatile session storage). [refer to `In-memory Mode`_]
73
+
74
+ * **Revertable**: Unlike traditional NoSQL stores, ``omni-json-db`` tracks internal states allowing you to unwrite (rollback a modification) or undelete a record. This provides a safety net similar to a manual "Undo" or a lightweight ACID rollback. [ref to `Rollback data`_]
75
+
76
+ * **Native CSV Support**: Built-in hooks for DictReader and DictWriter allow you to import massive datasets from CSV files or export your ``omni-json-db`` collections for analysis in Excel or Pandas.
77
+
78
+ * **Date-Based Lookups**: Every record is timestamped, enabling queries like "Give me all users modified last Tuesday." [refer to `Date Lookups`_]
79
+
80
+ 📌 Supported Python Versions
81
+ ****************************
82
+
83
+ ``omni-json-db`` has been tested with Python 3.7 - 3.14.
84
+
85
+ 🛠️ Quick Start
86
+ **************
87
+
88
+ Installation
89
+ ------------
90
+
91
+ .. code-block:: bash
92
+
93
+ pip install omni-json-db
94
+
95
+ Basic usage
96
+ -----------
97
+
98
+ .. code-block:: python
99
+
100
+ from omni_json_db import JDb
101
+ # Initialize the database from file
102
+ # Key-Value is Json+Json without compression
103
+ jdb = JDb("example.jdb")
104
+
105
+ # Store data
106
+ jdb["user:001"] = {"name" : "Ryan", "role": "Developer"}
107
+
108
+ # Retrieve data
109
+ user = jdb["user:001"]
110
+ print(user["name"]) # Output: Ryan
111
+
112
+ All standard ``dict`` methods work: ``keys()``, ``values()``, ``items()``, ``pop()``, ``setdefault()``, ``update()``.
113
+
114
+ In-memory Mode
115
+ --------------
116
+
117
+ .. code-block:: python
118
+
119
+ from omni_json_db import JDb
120
+ # Initialize the database in memory
121
+ # Key-Value is Json+Msgpack with Gzip compression
122
+ jdb = JDb(data_type="J+S", zip_type="gz")
123
+
124
+ # Store data
125
+ jdb += {"user:001" : {"name" : "Joe", "role": "Senior Developer"}}
126
+
127
+ # Retrieve data
128
+ user = jdb["user:001"]
129
+ print(user["name"]) # Output: Joe
130
+
131
+ # create 2nd JDb with same memory
132
+ jdb2 = JDb(jdb)
133
+
134
+ # Store data
135
+ jdb2["user:002"] ={"name" : "Kathy", "role": "CEO"}
136
+
137
+ assert jdb == jdb2
138
+ assert len(jdb) == 2
139
+ print(jdb["user:002"]["name"]) # Output: Kathy
140
+ print(set(jdb)) # Output: {'user:001', 'user:002'}
141
+
142
+ Rollback data
143
+ -------------
144
+
145
+ .. code-block:: python
146
+
147
+ from omni_json_db import JDb
148
+ # Initialize the database from file
149
+ # Key-Value is Json+Pickle with zstandard compression
150
+ jdb = JDb("fruit.jdb", data_type="J+P(zs)")
151
+
152
+ jdb["apple"] = "red"
153
+ jdb["apple"] = "blue" # modify
154
+ jdb.revert("apple") # unmodify
155
+ assert jdb["apple"] == 'red'
156
+
157
+ del jdb["apple"]
158
+ assert "apple" not in jdb
159
+
160
+ jdb.revert("apple") # unremove
161
+ assert jdb["apple"] == "red"
162
+
163
+ Query data
164
+ ----------
165
+
166
+ .. code-block:: python
167
+
168
+ from omni_json_db import JDb
169
+ # Initialize the database in memory
170
+ # Key-Value is Json+Marshal with no compression
171
+ jdb = JDb(data_type="J+M")
172
+
173
+ # insert value without key
174
+ jdb += [{'name': 'John', 'age': 22}, {'name': 'John', 'age': 37}, \
175
+ {'name': 'Bob', 'age': 42}, {'name': 'Megan', 'age': 27}]
176
+
177
+ print(jdb[:]) # print all records from jdb
178
+
179
+ matches = jdb.find(FUNC=lambda k,v: v.get('name', '') == 'John')
180
+ print(matches) # Output : {'0': {'name': 'John', 'age': 22}, '1': {'name': 'John', 'age': 37}}
181
+
182
+ matches = jdb.find(RE='John|Bob')
183
+ print(matches) # {'0': {'name': 'John', 'age': 22}, '1': {'name': 'John', 'age': 37}, '2': {'name': 'Bob', 'age': 42}}
184
+
185
+ Operator
186
+ --------
187
+
188
+ .. code-block:: python
189
+
190
+ from omni_json_db import JDb
191
+ # Initialize the database in memory
192
+ # Key+Value is Msgpack+Msgpack with lz4 compression
193
+ jdb = JDb(data_type="S+S(lz)")
194
+
195
+ # [1] KEY+VAL operators
196
+ # <jdb += data> == jdb.update(data)
197
+ data = {f'key{v}':v for v in range(100)}
198
+ jdb += data
199
+ assert len(jdb) == 100
200
+
201
+ # <jdb == data>
202
+ assert jdb == data
203
+
204
+ # <jdb |= ..> == jdb.insert(..)
205
+ jdb |= {f'key{v}':v+1 for v in range(102)}
206
+ assert len(jdb) == 102
207
+ assert jdb['key100'] == 101
208
+ assert jdb[-2.:] == {'key100':101, 'key101':102} # get last two modified records
209
+ assert jdb[(f'key{v}' for v in range(100))] == data # same as jdb[data] == data
210
+
211
+ # <jdb -= ..> == jdb.remove(..)
212
+ jdb -= ['key100', 'key101', 'key102', 'key103']
213
+ assert len(jdb) == 100
214
+ assert jdb == data
215
+
216
+ # <jdb &= ..> == jdb.replace(..)
217
+ jdb &= {f'key{v}':v+1 for v in range(200)}
218
+ assert len(jdb) == 100
219
+ assert jdb == {f'key{v}':v+1 for v in range(100)}
220
+
221
+ # <jdb ^= ..> == jdb.unmodify(..)
222
+ jdb ^= {f'key{v}' for v in range(100)} # same as jdb ^= data
223
+ assert len(jdb) == 100
224
+ assert jdb == data
225
+
226
+ # <jdb[:] = ..> == jdb.update(..)
227
+ jdb[:] = 0 # set all records to zero
228
+ assert len(jdb) == 100
229
+ assert jdb == {f'key{v}':0 for v in range(100)}
230
+ assert jdb.find(NE=0) == {}
231
+
232
+ # remove all records
233
+ jdb -= jdb # same as del jdb[:]
234
+ assert len(jdb) == 0
235
+
236
+ # <jdb ^= ..> == jdb.unremove(..)
237
+ jdb ^= {f'key{v}' for v in range(100)} # same as jdb ^= data
238
+ assert len(jdb) == 100
239
+ assert all(val == 0 for key,val in jdb.items())
240
+
241
+ # lambda VALUE operation
242
+ jdb[:] = lambda key,val: int(key.replace('key', '')) + val
243
+ assert jdb == data
244
+
245
+ # <del jdb[..]> == jdb.remove_fast(..)
246
+ del jdb[data] # same as del jdb[:]
247
+ assert len(jdb) == 0
248
+
249
+ # unremove all data
250
+ jdb ^= data
251
+ assert jdb == data
252
+
253
+ # <jdb[..]> == jdb.get_n(..) or jdb.get_all()
254
+ matches = jdb[('key2', 'key22', 'key44', 'key111')]
255
+ assert matches == {'key2':2, 'key22':22, 'key44':44}
256
+
257
+ # lambda KEY operation
258
+ matches = jdb[lambda key:key.endswith('1')]
259
+ assert set(matches) == {'key1', 'key11', 'key21', 'key31', 'key41', 'key51', 'key61', 'key71', 'key81', 'key91'}
260
+
261
+ # set all matched records to -1
262
+ jdb[matches] = -1
263
+ matches_2 = jdb[lambda key,val: val == -1]
264
+ assert set(matches) == set(matches_2)
265
+ assert matches_2 == jdb.find(EQ=-1)
266
+ assert matches_2 == jdb.find(FUNC=lambda val: val == -1)
267
+
268
+ # RE search
269
+ matches_3 = jdb[::r'1$']
270
+ assert matches_2 == matches_3
271
+
272
+ # unmodify
273
+ jdb ^= matches
274
+ assert jdb == data
275
+
276
+ # [2] KEY operators
277
+ # <jdb & {..}> == jdb.intersection(..)
278
+ matches = jdb & {f'key{v}' for v in range(98, 120)}
279
+ assert matches == {'key98', 'key99'}
280
+
281
+ # <{..} & jdb> == {..}.intersection(jdb)
282
+ matches_2 = {f'key{v}' for v in range(98, 120)} & jdb
283
+ assert matches == matches_2
284
+
285
+ # <jdb | {..}> == jdb.union(..)
286
+ matches = jdb | {f'key{v}' for v in range(10, 120)}
287
+ assert len(matches) == 120
288
+ assert matches == {f'key{v}' for v in range(0, 120)}
289
+
290
+ # <{..} | jdb> == {..}.union(jdb)
291
+ matches_2 = {f'key{v}' for v in range(10, 120)} | jdb
292
+ assert matches == matches_2
293
+
294
+ # <jdb + {..}> == jdb.union(..)
295
+ matches = jdb + {f'key{v}' for v in range(10, 120)}
296
+ assert matches == matches_2
297
+
298
+ # <{..} + jdb> == {..}.union(jdb)
299
+ matches_2 = {f'key{v}' for v in range(10, 120)} + jdb
300
+ assert matches == matches_2
301
+
302
+ # <jdb - {..}> == jdb.difference(..)
303
+ matches = jdb - {f'key{v}' for v in range(0, 98)}
304
+ assert matches == {'key98', 'key99'}
305
+
306
+ # <{..} - jdb> == {..}.difference(jdb)
307
+ matches = {f'key{v}' for v in range(2, 102)} - jdb
308
+ assert matches == {'key100', 'key101'}
309
+
310
+ # <jdb ^ {..}> == jdb.non_intersection(..)
311
+ matches = jdb ^ {f'key{v}' for v in range(1, 101)}
312
+ assert matches == {'key0', 'key100'}
313
+
314
+ # <{..} ^ jdb> == {..}.non_intersection(jdb)
315
+ matches_2 = {f'key{v}' for v in range(1, 101)} ^ jdb
316
+ assert matches == matches_2
317
+
318
+ # <.. in jdb> == jdb.has_all(..)
319
+ assert 'key10' in jdb
320
+ assert {'key10', 'key90'} in jdb
321
+ assert {'key10', 'key90', 'key110', 'key190'} not in jdb
322
+ assert jdb.has('key10')
323
+ assert jdb.has_all('key10')
324
+ assert jdb.has_any('key10')
325
+ assert jdb.has_all({'key10', 'key90'})
326
+ assert jdb.has_any({'key10', 'key90', 'key110', 'key190'})
327
+ assert jdb.is_disjoint({'key110', 'key190'})
328
+
329
+ Date Lookups
330
+ ------------
331
+
332
+ .. code-block:: python
333
+
334
+ from omni_json_db import JDb
335
+ import datetime as dt
336
+
337
+ # Initialize the database in memory
338
+ # Key+Value is Json+Msgpack with Brotli compression
339
+ # using BTree as Key Table for better memory usage
340
+ jdb = JDb(data_type="J+S(br)", key_limit="bt")
341
+
342
+ # insert data
343
+ fruits = {'apple':'red', 'banana':'yellow', 'mango':'yellow', 'lemon':'yellow', 'tomato':'red'}
344
+ jdb += fruits
345
+
346
+ # datetime for create date, date for modify date
347
+ now = dt.datetime.now()
348
+ today = now.date()
349
+
350
+ # find create date: date == now
351
+ matches = jdb[now]
352
+ assert matches == fruits
353
+
354
+ # find create date: date >= now
355
+ matches = jdb[now:]
356
+ assert matches == fruits
357
+
358
+ # find create date: date < now
359
+ matches = jdb[:now]
360
+ assert len(matches) == 0
361
+
362
+ # find create date: now <= date <= now+1
363
+ next_date = now + dt.timedelta(days=1)
364
+ matches = jdb[now:next_date]
365
+ assert matches == fruits
366
+
367
+ prev_date = now - dt.timedelta(days=1)
368
+ prev_week = now - dt.timedelta(days=7)
369
+
370
+ # change key create date
371
+ jdb.keys['apple', 'tomato'] = prev_date
372
+ jdb.keys['mango'] = prev_week
373
+ assert jdb[prev_date] == {'apple':'red', 'tomato':'red'}
374
+ assert jdb[prev_week] == {'mango':'yellow'}
375
+
376
+ # find create date: date == now
377
+ matches = jdb[now]
378
+ assert set(matches) == {'banana', 'lemon'}
379
+
380
+ # find create date: date < now
381
+ matches = jdb[:now]
382
+ assert set(matches) == {'apple', 'mango', 'tomato'}
383
+
384
+ # find modify date: date == today
385
+ matches = jdb[today]
386
+ assert matches == fruits
387
+
388
+ # change key modify date + create date
389
+ new_modify_date = prev_date.date()
390
+ new_create_date = prev_week.date()
391
+ assert new_modify_date >= new_create_date
392
+ jdb.keys['lemon'] = f'{new_modify_date} {new_create_date}'
393
+
394
+ # find modify date: date == today
395
+ matches = jdb[today]
396
+ assert set(matches) == {'apple', 'banana', 'mango', 'tomato'}
397
+
398
+ # find modify date: date == prev_date
399
+ matches = jdb[prev_date.date()]
400
+ assert set(matches) == {'lemon'}
401
+
402
+ # change all keys create date
403
+ jdb.keys[:] = today
404
+ assert jdb[today] == fruits
405
+
406
+ Advanced Usage
407
+ --------------
408
+
409
+ .. code-block:: python
410
+
411
+ from omni_json_db import JDb
412
+
413
+ jdb = JDb()
414
+
415
+ fruits = {'apple':'red', 'banana':'yellow', 'mango':'yellow', 'lemon':'yellow', 'tomato':'red'}
416
+
417
+ # insert records
418
+ with jdb.open() as fp:
419
+ for fruit,color in fruits.items():
420
+ jdb._write(fp, fruit, color)
421
+
422
+ assert jdb == fruits
423
+
424
+ # modify records
425
+ with jdb.open() as fp:
426
+ for fruit in fruits:
427
+ color = jdb._read(fp, fruit)
428
+ jdb._write(fp, fruit, color.upper())
429
+
430
+ assert jdb != fruits
431
+ assert set(jdb) == set(fruits)
432
+
433
+ # unmodify records
434
+ with jdb.open() as fp:
435
+ for fruit in fruits:
436
+ jdb._unwrite(fp, fruit)
437
+
438
+ assert jdb == fruits
439
+
440
+ # remove records
441
+ with jdb.open() as fp:
442
+ for fruit in fruits:
443
+ jdb._delete(fp, fruit)
444
+
445
+ assert len(jdb) == 0
446
+
447
+ # unremove records
448
+ with jdb.open() as fp:
449
+ for fruit in fruits:
450
+ jdb._undelete(fp, fruit)
451
+
452
+ assert jdb == fruits
453
+
454
+ #---------------------------------------
455
+ with jdb.open() as fp:
456
+ key_table = jdb.key_table
457
+
458
+ # replace
459
+ for fruit in key_table:
460
+ color = jdb._read(fp, fruit)
461
+ jdb._write(fp, fruit, color.upper())
462
+
463
+ # unmodify
464
+ for fruit in key_table:
465
+ jdb._unwrite(fp, fruit)
466
+
467
+ # remove
468
+ for fruit in fruits:
469
+ jdb._delete(fp, fruit)
470
+
471
+ # unremove
472
+ for fruit in fruits:
473
+ jdb._undelete(fp, fruit)
474
+
475
+ assert jdb == fruits
476
+
477
+ #---------------------------------------
478
+ # replace all
479
+ jdb[:] = lambda k,v: v.upper()
480
+
481
+ # unmodify all
482
+ jdb ^= jdb
483
+
484
+ # remove all
485
+ jdb -= jdb
486
+
487
+ # unremove all
488
+ jdb ^= fruits
489
+
490
+ assert jdb == fruits
491
+
492
+ Network Mode
493
+ -------------
494
+
495
+ **Server side:**
496
+
497
+ .. code-block:: python
498
+
499
+ >> from omni_json_db import run_files_server
500
+ >> run_files_server(host='0.0.0.0', port=59698, files='net_storage.jdb')
501
+
502
+ **Client side:**
503
+
504
+ .. code-block:: python
505
+
506
+ >> from omni_json_db import JDb, JNetFiles
507
+ >> jdb = JDb(JNetFiles(('0.0.0.0', 59898)))
508
+
509
+ 📝 Specifications
510
+ *****************
511
+
512
+ Supported Data Formats
513
+ ----------------------
514
+
515
+ Configure ``data_type`` during initialization:
516
+
517
+ * ``J+J``: JSON Key + JSON Value (default)
518
+ * ``J+S``: JSON Key + MsgPack Value
519
+ * ``J+M``: JSON Key + Marshal Value
520
+ * ``J+P``: JSON Key + Pickle Value
521
+ * ``S+J``: MsgPack Key + JSON Value
522
+ * ``S+S``: MsgPack Key + MsgPack Value
523
+ * ``S+M``: MsgPack Key + Marshal Value
524
+ * ``S+P``: MsgPack Key + Pickle Value
525
+
526
+ **Data size = 70,840,580 (MB = 1,000,000B, no zip)**
527
+
528
+ +-------------------+------------+-------+----------+-----------+----------------+---------------+
529
+ | ``data_type`` | size | ratio | read | write | GOODs | BADs |
530
+ +===================+============+=======+==========+===========+================+===============+
531
+ | ``J+J`` or ``S+J``| 70,840,580 | 1.00 | 75.3MB/s | 358.0MB/s |* fastest write |* no set() |
532
+ | | | | | |* faster read |* no tuple() |
533
+ | | | | | |* readable |* weak bytes |
534
+ +-------------------+------------+-------+----------+-----------+----------------+---------------+
535
+ | ``J+S`` or ``S+S``| 47,616,008 | 1.48 | 77.4MB/s | 354.2MB/s |* smallest size |* no tuple() |
536
+ | | | | | |* faster read |* unreadable |
537
+ | | | | | |* faster write | |
538
+ +-------------------+------------+-------+----------+-----------+----------------+---------------+
539
+ | ``J+M`` or ``S+M``| 72,430,958 | 0.97 | 81.4MB/s | 177.1MB/s |* all type [1]_ |* biggest size |
540
+ | | | | | |* fastest read |* unreadable |
541
+ +-------------------+------------+-------+----------+-----------+----------------+---------------+
542
+ | ``J+P`` or ``S+P``| 70,207,207 | 1.01 | 64.9MB/s | 22.8MB/s |* all type [1]_ |* slowest read |
543
+ | | | | | | |* slowest write|
544
+ | | | | | | |* unreadable |
545
+ +-------------------+------------+-------+----------+-----------+----------------+---------------+
546
+
547
+ .. [1] all type = ``str``, ``bytes``, ``bool``, ``int``, ``float``, ``list``, ``tuple``, ``set``, ``dict``, ``None``
548
+
549
+ Supported Zip Formats
550
+ ---------------------
551
+
552
+ Configure ``zip_type`` during initialization:
553
+
554
+ * ``no``: no compression for Value (default)
555
+ * ``gz``: Gzip (mode=9) compression for Value
556
+ * ``bz``: Bzip2 (mode=9) compression for Value
557
+ * ``xz``: LZMA compression for Value
558
+ * ``zs``: Zstandard (mode=22) compression for Value
559
+ * ``br``: Brotli (mode=6) compression for Value (better than ``gz``)
560
+ * ``z1``: Zstandard (mode=6) compression for Value (better than ``gz``)
561
+ * ``z2``: Zstandard (mode=11) compression for Value
562
+ * ``lz``: LZ4 (mode=0) compression for Value
563
+
564
+ **Data size = 70,840,580 (MB = 1,000,000B)**
565
+
566
+ +------------+------------+-------+----------+-----------+---------------+---------------+
567
+ |``zip_type``| size | ratio | read | write | GOODs | BADs |
568
+ +============+============+=======+==========+===========+===============+===============+
569
+ | ``no`` | 70,840,580 | 1.00 | 75.3MB/s | 358.0MB/s |* fastest speed|* biggest size |
570
+ +------------+------------+-------+----------+-----------+---------------+---------------+
571
+ | ``gz`` | 16,915,844 | 4.18 | 65.5MB/s | 5.1MB/s | |* slower zip |
572
+ +------------+------------+-------+----------+-----------+---------------+---------------+
573
+ | ``bz`` | 11,394,042 | 6.21 | 26.4MB/s | 10.8MB/s |* better ratio |* slowest unzip|
574
+ +------------+------------+-------+----------+-----------+---------------+---------------+
575
+ | ``xz`` | 11,340,548 | 6.24 | 54.9MB/s | 2.3MB/s |* better ratio |* slower zip |
576
+ | | | | | | |* slower unzip |
577
+ +------------+------------+-------+----------+-----------+---------------+---------------+
578
+ | ``zs`` | 11,119,665 | 6.37 | 73.0MB/s | 1.7MB/s |* best ratio |* slowest zip |
579
+ | | | | | |* faster unzip | |
580
+ +------------+------------+-------+----------+-----------+---------------+---------------+
581
+ | ``br`` | 13,700,696 | 5.17 | 65.8MB/s | 25.3MB/s |* better ``gz``| |
582
+ +------------+------------+-------+----------+-----------+---------------+---------------+
583
+ | ``z1`` | 14,738,859 | 4.80 | 73.6MB/s | 70.8MB/s |* faster zip | |
584
+ | | | | | |* faster unzip | |
585
+ +------------+------------+-------+----------+-----------+---------------+---------------+
586
+ | ``z2`` | 13,799,407 | 5.13 | 72.7MB/s | 23.6MB/s |* faster unzip | |
587
+ +------------+------------+-------+----------+-----------+---------------+---------------+
588
+ | ``lz`` | 26,226,039 | 2.70 | 75.6MB/s | 202.4MB/s |* fastest zip |* worst ratio |
589
+ | | | | | |* fastest unzip| |
590
+ +------------+------------+-------+----------+-----------+---------------+---------------+
591
+
592
+ Supported Key Table Formats
593
+ ---------------------------
594
+
595
+ Configure ``key_limit`` during initialization:
596
+
597
+ * ``no``: ``dict`` for key_table (default)
598
+ * ``bt``: ``BTree`` for key_table (save 44.3% vs ``dict``)
599
+ * ``l0`` - ``l5``: ``LiteKeyTable`` modes (save 60-75%+ vs ``dict``)
600
+
601
+ **Table size = 3,241,854 keys**
602
+
603
+ +---------------+--------+--------------+------------+--------------+
604
+ | ``key_limit`` | memory | key search | HIT > get()| MISS > get() |
605
+ +===============+========+==============+============+==============+
606
+ | ``no`` | 519MB | 48.59Mo/s | 29.28Mo/s | 18.3Mo/s |
607
+ +---------------+--------+--------------+------------+--------------+
608
+ | ``bt`` | 289MB | 3.46Mo/s | 3.07Mo/s | 8.04Mo/s |
609
+ +---------------+--------+--------------+------------+--------------+
610
+ | ``l3`` | 85MB | 2.01Mo/s | 2.01Mo/s | 1.59Mo/s |
611
+ +---------------+--------+--------------+------------+--------------+
612
+
613
+ 📊 Benchmarking
614
+ ***************
615
+
616
+ Testing
617
+ -------
618
+
619
+ .. code-block:: python
620
+
621
+ >> from omni_json_db import JDb
622
+ >> size = 1_000_000
623
+ >> jdb = JDb(data_type='J+J')
624
+ >> data = {f'key{k}':k for k in range(size)}
625
+
626
+ >> # Benchmarking operations
627
+ >> jdb += data # insert
628
+ >> jdb[:] # get_all
629
+ >> jdb -= data # remove
630
+ >> jdb ^= data # revert=unremove
631
+ >> jdb[data] = -1 # replace
632
+ >> jdb ^= data # revert=unmodify
633
+ >> print(jdb == data) # Output: True
634
+
635
+ Results
636
+ -------
637
+
638
+ +-------+---------+---------+---------+----------+---------+----------+
639
+ | size | insert | get_all | remove | unremove | replace | unmodify |
640
+ +=======+=========+=========+=========+==========+=========+==========+
641
+ | 1 | 132 μs | 89 μs | 111 μs | 96 μs | 91 μs | 83 μs |
642
+ +-------+---------+---------+---------+----------+---------+----------+
643
+ | 10 | 136 μs | 93 μs | 142 μs | 145 μs | 183 μs | 177 μs |
644
+ +-------+---------+---------+---------+----------+---------+----------+
645
+ | 100 | 442 μs | 319 μs | 594 μs | 680 μs | 876 μs | 976 μs |
646
+ +-------+---------+---------+---------+----------+---------+----------+
647
+ | 1K | 3.37 ms | 2.71 ms | 5.24 ms | 5.9 ms | 7.61 ms | 9.12 ms |
648
+ +-------+---------+---------+---------+----------+---------+----------+
649
+ | 10K | 32.2 ms | 26 ms | 54.3 ms | 55.8 ms | 77.5 ms | 91.1 ms |
650
+ +-------+---------+---------+---------+----------+---------+----------+
651
+ | 100K | 358 ms | 262 ms | 626 ms | 583 ms | 774 ms | 930 ms |
652
+ +-------+---------+---------+---------+----------+---------+----------+
653
+ | 1M | 3.87 s | 2.78 s | 7 s | 6.09 s | 8.15 s | 9.83 s |
654
+ +-------+---------+---------+---------+----------+---------+----------+
655
+
656
+ 👥 Contributing
657
+ ***************
658
+
659
+ Whether reporting bugs, discussing improvements and new ideas or writing extensions: Contributions to ``omni-json-db`` are welcome! Here's how to get started:
660
+
661
+ 1. Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug.
662
+ 2. Fork `the repository <https://github.com/lukatrum/omni-json-db/>`_ on Github, create a new branch off the ``master`` branch and start making your changes (known as `GitHub Flow <https://docs.github.com/en/get-started/using-github/github-flow>`_).
663
+ 3. Write a test which shows that the bug was fixed or that the feature works as expected.
664
+ 4. Send a pull request and bug the maintainer until it gets merged and published ☺
665
+