objbase 0.3.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- objbase/__init__.py +59 -0
- objbase/asyncio/__init__.py +0 -0
- objbase/asyncio/async_file_storage.py +40 -0
- objbase/asyncio/async_inventory.py +47 -0
- objbase/asyncio/async_mongodb_storage.py +46 -0
- objbase/asyncio/async_redis_storage.py +36 -0
- objbase/asyncio/async_sqlite_storage.py +19 -0
- objbase/asyncio/async_storage.py +18 -0
- objbase/asyncio/threaded_storage.py +30 -0
- objbase/errors.py +11 -0
- objbase/interface.py +27 -0
- objbase/inventory.py +59 -0
- objbase/py.typed +0 -0
- objbase/pydantic.py +123 -0
- objbase/storage/__init__.py +0 -0
- objbase/storage/file_storage.py +268 -0
- objbase/storage/inmemory_storage.py +51 -0
- objbase/storage/mongodb_storage.py +42 -0
- objbase/storage/redis_storage.py +68 -0
- objbase/storage/sqlite_storage.py +79 -0
- objbase/util/__init__.py +0 -0
- objbase/util/file_util.py +102 -0
- objbase/util/mongodb_util.py +45 -0
- objbase/util/redis_util.py +35 -0
- objbase-0.3.0.dist-info/METADATA +780 -0
- objbase-0.3.0.dist-info/RECORD +28 -0
- objbase-0.3.0.dist-info/WHEEL +4 -0
- objbase-0.3.0.dist-info/licenses/LICENSE +21 -0
|
@@ -0,0 +1,780 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: objbase
|
|
3
|
+
Version: 0.3.0
|
|
4
|
+
Summary: Damn simple object store for Python dicts and Pydantic models across multiple backends
|
|
5
|
+
Project-URL: Homepage, https://github.com/fm-labs/objbase
|
|
6
|
+
Project-URL: Issues, https://github.com/fm-labs/objbase/issues
|
|
7
|
+
Author-email: fm-labs <code@fmlabs.dev>
|
|
8
|
+
License-Expression: MIT
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Keywords: fastapi,key-value,mongodb,pydantic,redis,sqlite,storage
|
|
11
|
+
Classifier: Development Status :: 3 - Alpha
|
|
12
|
+
Classifier: Framework :: AsyncIO
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: Operating System :: OS Independent
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
18
|
+
Classifier: Topic :: Database
|
|
19
|
+
Classifier: Typing :: Typed
|
|
20
|
+
Requires-Python: >=3.13
|
|
21
|
+
Provides-Extra: all
|
|
22
|
+
Requires-Dist: pydantic>=2.8; extra == 'all'
|
|
23
|
+
Requires-Dist: pymongo>=4.13; extra == 'all'
|
|
24
|
+
Requires-Dist: redis>=6.0; extra == 'all'
|
|
25
|
+
Provides-Extra: mongodb
|
|
26
|
+
Requires-Dist: pymongo>=4.13; extra == 'mongodb'
|
|
27
|
+
Provides-Extra: pydantic
|
|
28
|
+
Requires-Dist: pydantic>=2.8; extra == 'pydantic'
|
|
29
|
+
Provides-Extra: redis
|
|
30
|
+
Requires-Dist: redis>=6.0; extra == 'redis'
|
|
31
|
+
Description-Content-Type: text/markdown
|
|
32
|
+
|
|
33
|
+
# InventoryDB
|
|
34
|
+
|
|
35
|
+
Damn simple object store for Python dicts and Pydantic models across multiple backends
|
|
36
|
+
(in-memory, file-based, SQLite, Redis, MongoDB, and more).
|
|
37
|
+
Provides a minimal API for storing, retrieving, updating, and deleting objects.
|
|
38
|
+
|
|
39
|
+
__No thrills__ - **just a simple key-value store for serializable Python objects, with a consistent API across different storage backends.**
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
## What you get
|
|
43
|
+
|
|
44
|
+
- Basic CRUD operations: `save`, `get`, `filter`, `keys`, `patch`, `delete`
|
|
45
|
+
- Multiple storage adapters (in-memory, file-based, SQLite, Redis, MongoDB)
|
|
46
|
+
- Optional Pydantic model validation with `PydanticInventory` / `AsyncPydanticInventory`
|
|
47
|
+
- Async support via `AsyncInventory` with async storage adapters (in-memory, file-based, SQLite, Redis, MongoDB)
|
|
48
|
+
- Easy FastAPI integration with dependency injection
|
|
49
|
+
- Fully typed (ships `py.typed`), checked with `mypy --strict`
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
---
|
|
53
|
+
|
|
54
|
+
## Installation
|
|
55
|
+
|
|
56
|
+
Requires Python 3.13+. The core package has no dependencies; in-memory, file-based
|
|
57
|
+
and SQLite storage work out of the box. Install extras for the other backends:
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
pip install objbase # core only
|
|
61
|
+
pip install "objbase[redis]" # + redis-py, for (Async)RedisInventoryStorage
|
|
62
|
+
pip install "objbase[mongodb]" # + pymongo, for (Async)MongoDBInventoryStorage
|
|
63
|
+
pip install "objbase[pydantic]" # + pydantic, for (Async)PydanticInventory
|
|
64
|
+
pip install "objbase[all]" # everything
|
|
65
|
+
# or with uv
|
|
66
|
+
uv add "objbase[redis]"
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
---
|
|
70
|
+
|
|
71
|
+
## Quick Start
|
|
72
|
+
|
|
73
|
+
Every item must have an `"id"` field. Use `Inventory` with any storage adapter:
|
|
74
|
+
|
|
75
|
+
```python
|
|
76
|
+
from objbase import Inventory, InMemoryInventoryStorage
|
|
77
|
+
|
|
78
|
+
storage = InMemoryInventoryStorage()
|
|
79
|
+
todos = Inventory(item_type="todo", storage=storage)
|
|
80
|
+
|
|
81
|
+
todos.save({"id": "1", "title": "Buy milk", "done": False})
|
|
82
|
+
todos.save({"id": "2", "title": "Walk dog", "done": False})
|
|
83
|
+
|
|
84
|
+
todos.get("1") # → {"id": "1", "title": "Buy milk", "done": False}
|
|
85
|
+
todos.filter() # → [{"id": "1", ...}, {"id": "2", ...}]
|
|
86
|
+
todos.keys() # → ["1", "2"] (order unspecified)
|
|
87
|
+
todos.patch("1", {"done": True}) # → {"id": "1", ..., "done": True}
|
|
88
|
+
todos.delete("1") # → True
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
Swapping the backend requires only changing the `storage` argument — the `Inventory`
|
|
92
|
+
API stays identical.
|
|
93
|
+
|
|
94
|
+
All public classes can be imported from the top-level `objbase` package, as above,
|
|
95
|
+
or from their submodules (e.g. `objbase.storage.sqlite_storage`) as in the examples below.
|
|
96
|
+
|
|
97
|
+
### Behaviour
|
|
98
|
+
|
|
99
|
+
All adapters follow the same contract (verified by a shared test suite):
|
|
100
|
+
|
|
101
|
+
- `get` returns `None` for a missing item; `filter` and `keys` return `[]` for an empty type.
|
|
102
|
+
- `save` inserts a new item or **replaces** an existing one entirely (it does not merge fields).
|
|
103
|
+
- `patch` merges the given fields into an existing item. It cannot change the item's `id`.
|
|
104
|
+
- `delete` returns `True` if the item was removed, `False` if it did not exist.
|
|
105
|
+
- Items you get back are copies; mutating them does not change stored data.
|
|
106
|
+
|
|
107
|
+
Errors are raised, not returned:
|
|
108
|
+
|
|
109
|
+
| Situation | Exception |
|
|
110
|
+
|---|---|
|
|
111
|
+
| `save` an item without an `id` | `ValueError` |
|
|
112
|
+
| `patch` a missing item | `objbase.errors.ItemNotFoundError` (a `LookupError`) |
|
|
113
|
+
| The storage backend reports a failed write | `objbase.errors.InventoryError` |
|
|
114
|
+
|
|
115
|
+
---
|
|
116
|
+
|
|
117
|
+
## Storage Adapters
|
|
118
|
+
|
|
119
|
+
| Adapter | Sync class | Async class | When to use |
|
|
120
|
+
|---|---|---|---|
|
|
121
|
+
| In-Memory | `InMemoryInventoryStorage` | same class | Testing / prototyping — volatile |
|
|
122
|
+
| File (one file per type) | `FileBasedInventoryStorage` | `AsyncFileBasedInventoryStorage` | Simple persistence for small datasets |
|
|
123
|
+
| File (one file per item) | `DirectoryBasedInventoryStorage` | `AsyncDirectoryBasedInventoryStorage` | Medium datasets; per-item file operations |
|
|
124
|
+
| SQLite | `SQLiteInventoryStorage` | `AsyncSQLiteInventoryStorage` | ACID persistence with zero external deps |
|
|
125
|
+
| Redis | `RedisInventoryStorage` | `AsyncRedisInventoryStorage` | High-performance / distributed access |
|
|
126
|
+
| MongoDB | `MongoDBInventoryStorage` | `AsyncMongoDBInventoryStorage` | Document-oriented storage and complex queries |
|
|
127
|
+
|
|
128
|
+
Each async adapter uses the same data layout as its sync counterpart, so both can
|
|
129
|
+
work on the same data. The file-based and SQLite async adapters run the sync code in
|
|
130
|
+
a worker thread (`asyncio.to_thread`) and need no extra dependencies; the Redis and
|
|
131
|
+
MongoDB ones use the drivers' native async clients.
|
|
132
|
+
|
|
133
|
+
### In-Memory
|
|
134
|
+
|
|
135
|
+
```python
|
|
136
|
+
from objbase.storage.inmemory_storage import InMemoryInventoryStorage
|
|
137
|
+
|
|
138
|
+
storage = InMemoryInventoryStorage()
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
No configuration needed. Data is lost when the process exits.
|
|
142
|
+
Also implements `AsyncInventoryStorage` — the async methods delegate to their
|
|
143
|
+
sync counterparts.
|
|
144
|
+
|
|
145
|
+
### File-Based (single file per type)
|
|
146
|
+
|
|
147
|
+
```python
|
|
148
|
+
from objbase.storage.file_storage import FileBasedInventoryStorage
|
|
149
|
+
|
|
150
|
+
storage = FileBasedInventoryStorage(base_dir="/var/data/myapp")
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
All items of one type are stored in `{base_dir}/{item_type}.json`.
|
|
154
|
+
The directory must exist before construction; type files are created on first write.
|
|
155
|
+
Item types must be safe file names, otherwise `ValueError` is raised (see [Path safety](#path-safety)).
|
|
156
|
+
|
|
157
|
+
Safe to use from multiple threads and processes on the same machine. Each write
|
|
158
|
+
holds an exclusive lock on a hidden `.{item_type}.json.lock` file while it reads,
|
|
159
|
+
changes and rewrites the type file, so concurrent writes are never lost. Files
|
|
160
|
+
are replaced atomically, so readers never see a half-written file and a crash
|
|
161
|
+
mid-write cannot corrupt data. Lock files are left in place after use.
|
|
162
|
+
|
|
163
|
+
Every write rewrites the whole type file, so this adapter suits small datasets.
|
|
164
|
+
Locks are advisory and may not work on network file systems (NFS, SMB).
|
|
165
|
+
|
|
166
|
+
`AsyncFileBasedInventoryStorage(base_dir=...)` is the async counterpart. It uses the
|
|
167
|
+
same files and locks, running each call in a worker thread (`asyncio.to_thread`),
|
|
168
|
+
so it can share a directory with the sync adapter.
|
|
169
|
+
|
|
170
|
+
### File-Based (one file per item)
|
|
171
|
+
|
|
172
|
+
```python
|
|
173
|
+
from objbase.storage.file_storage import DirectoryBasedInventoryStorage
|
|
174
|
+
|
|
175
|
+
storage = DirectoryBasedInventoryStorage(base_dir="/var/data/myapp")
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Items are stored at `{base_dir}/{item_type}/{id}.json`.
|
|
179
|
+
Type directories are created automatically on first write.
|
|
180
|
+
Item types and ids must be safe file names, otherwise `ValueError` is raised (see [Path safety](#path-safety)).
|
|
181
|
+
|
|
182
|
+
Each type directory also contains an index file, `.index`, listing the ids of
|
|
183
|
+
all items of that type, one per line. Writes append new ids and deletes remove
|
|
184
|
+
them, so `keys()` reads the index instead of scanning the directory. A type
|
|
185
|
+
directory without an index (e.g. data written by an older version) is scanned
|
|
186
|
+
until the next write or delete creates the index. If the index gets out of step
|
|
187
|
+
with the item files — after a crash between writing an item and updating the
|
|
188
|
+
index, or after editing item files by hand — call
|
|
189
|
+
`storage.rebuild_index(item_type)`. Ids and item types can't contain newlines.
|
|
190
|
+
|
|
191
|
+
Item files are replaced atomically, so readers never see a half-written item.
|
|
192
|
+
Writes and deletes hold an exclusive lock on `.index.lock` in the type
|
|
193
|
+
directory while they update the item file and the index, so the index stays
|
|
194
|
+
correct with concurrent writers across threads and processes; concurrent writes
|
|
195
|
+
to the same item are last-writer-wins. Locks are advisory and may not work on
|
|
196
|
+
network file systems (NFS, SMB).
|
|
197
|
+
|
|
198
|
+
`AsyncDirectoryBasedInventoryStorage(base_dir=...)` is the async counterpart. It uses
|
|
199
|
+
the same files, index and locks, running each call in a worker thread
|
|
200
|
+
(`asyncio.to_thread`); rebuild its index with `await storage.arebuild_index(item_type)`.
|
|
201
|
+
|
|
202
|
+
### Path safety
|
|
203
|
+
|
|
204
|
+
Both file-based adapters build file paths from item types (and, for
|
|
205
|
+
`DirectoryBasedInventoryStorage`, ids), so they guard against path traversal:
|
|
206
|
+
|
|
207
|
+
- Names must be a single path component: empty names, `.`, `..`, and names
|
|
208
|
+
containing `/`, `\`, NUL or newlines are rejected. On Windows, `< > : " | ? *`
|
|
209
|
+
and trailing dots or spaces are rejected too, since Windows would otherwise
|
|
210
|
+
treat `C:x` as a drive-relative path and `.. ` as `..`.
|
|
211
|
+
- Every path is resolved with `os.path.realpath` and must lie inside the base
|
|
212
|
+
directory. A symlink inside the base directory that points outside it (to a
|
|
213
|
+
type file, type directory, item file, index or lock file) makes the operation
|
|
214
|
+
raise `ValueError` instead of following it. Symlinks that stay inside the base
|
|
215
|
+
directory, and a base directory that is itself a symlink, work normally.
|
|
216
|
+
|
|
217
|
+
The check runs before each file operation, so it doesn't protect against an
|
|
218
|
+
attacker who can write to the base directory and swaps in a symlink between the
|
|
219
|
+
check and the operation. Don't give untrusted users write access to it.
|
|
220
|
+
|
|
221
|
+
### SQLite
|
|
222
|
+
|
|
223
|
+
```python
|
|
224
|
+
from objbase.storage.sqlite_storage import SQLiteInventoryStorage
|
|
225
|
+
|
|
226
|
+
storage = SQLiteInventoryStorage(db_path="myapp.db")
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
Uses a single `items` table with a `(item_type, id)` primary key and JSON
|
|
230
|
+
blob storage. The table is created automatically. No external dependencies needed.
|
|
231
|
+
`AsyncSQLiteInventoryStorage(db_path=...)` uses the same table, so sync and async adapters
|
|
232
|
+
can share a database. It runs each call in a worker thread (`asyncio.to_thread`), so it
|
|
233
|
+
also needs no extra dependencies.
|
|
234
|
+
|
|
235
|
+
### Redis
|
|
236
|
+
|
|
237
|
+
```python
|
|
238
|
+
import redis
|
|
239
|
+
from objbase.storage.redis_storage import RedisInventoryStorage
|
|
240
|
+
|
|
241
|
+
client = redis.Redis(host="localhost", port=6379, decode_responses=True)
|
|
242
|
+
storage = RedisInventoryStorage(redis_client=client)
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
Each item type is one Redis hash, `inventory:{item_type}`, mapping item ids to
|
|
246
|
+
JSON-encoded items, so value types (numbers, booleans, lists, nested dicts) are
|
|
247
|
+
preserved. Pass `key_prefix="myapp:"` to use a different prefix than `inventory:`.
|
|
248
|
+
|
|
249
|
+
Pass a pre-configured `redis.Redis` client (sync); `decode_responses` may be on or off.
|
|
250
|
+
Requires `redis-py`. `AsyncRedisInventoryStorage` takes a `redis.asyncio.Redis` client
|
|
251
|
+
and uses the same layout, so sync and async adapters can share data.
|
|
252
|
+
|
|
253
|
+
### MongoDB
|
|
254
|
+
|
|
255
|
+
```python
|
|
256
|
+
import pymongo
|
|
257
|
+
from objbase.storage.mongodb_storage import MongoDBInventoryStorage
|
|
258
|
+
|
|
259
|
+
client = pymongo.MongoClient("mongodb://localhost:27017")
|
|
260
|
+
storage = MongoDBInventoryStorage(mongo_client=client)
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
Items are stored in the `inventory` database, one collection per `item_type`.
|
|
264
|
+
The MongoDB `_id` field is stripped from results automatically.
|
|
265
|
+
Pass a pre-configured `pymongo.MongoClient`. Requires `pymongo`. `AsyncMongoDBInventoryStorage`
|
|
266
|
+
takes a `pymongo.AsyncMongoClient` and uses the same layout, so sync and async adapters
|
|
267
|
+
can share data. Both accept an optional MongoDB `query` in `items` / `aitems` to filter results.
|
|
268
|
+
|
|
269
|
+
---
|
|
270
|
+
|
|
271
|
+
## Pydantic Models
|
|
272
|
+
|
|
273
|
+
Use `PydanticInventory` to validate items against a Pydantic `BaseModel`.
|
|
274
|
+
`save` and `get` return typed model instances instead of plain dicts.
|
|
275
|
+
|
|
276
|
+
```python
|
|
277
|
+
from pydantic import BaseModel
|
|
278
|
+
from objbase.pydantic import PydanticInventory
|
|
279
|
+
from objbase.storage.inmemory_storage import InMemoryInventoryStorage
|
|
280
|
+
|
|
281
|
+
|
|
282
|
+
class Todo(BaseModel):
|
|
283
|
+
id: str
|
|
284
|
+
title: str
|
|
285
|
+
done: bool = False
|
|
286
|
+
|
|
287
|
+
|
|
288
|
+
todos = PydanticInventory(
|
|
289
|
+
item_type="todo",
|
|
290
|
+
storage=InMemoryInventoryStorage(),
|
|
291
|
+
model_class=Todo,
|
|
292
|
+
)
|
|
293
|
+
|
|
294
|
+
todos.save(Todo(id="1", title="Buy milk"))
|
|
295
|
+
item = todos.get("1") # returns a Todo instance (or None), not a dict
|
|
296
|
+
if item is not None:
|
|
297
|
+
print(item.done) # False
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
The model type is inferred from `model_class`, so type checkers know that
|
|
301
|
+
`todos.get()` returns `Todo | None` and `todos.filter()` returns `list[Todo]`.
|
|
302
|
+
`todos.keys()` returns the item ids (`list[str]`) without loading or validating
|
|
303
|
+
any items.
|
|
304
|
+
|
|
305
|
+
`save` and `patch` validate the complete item before writing it. Data that fails
|
|
306
|
+
validation raises `pydantic.ValidationError` and is never stored:
|
|
307
|
+
|
|
308
|
+
```python
|
|
309
|
+
todos.patch("1", {"done": "not a bool"}) # raises ValidationError; item unchanged
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
Items are stored in their validated, JSON-compatible form (`model_dump(mode="json")`),
|
|
313
|
+
so values Pydantic coerces are stored normalized: patching `{"done": "true"}` stores `True`.
|
|
314
|
+
|
|
315
|
+
### Async
|
|
316
|
+
|
|
317
|
+
`AsyncPydanticInventory` has the same methods and behaviour, as coroutines, and
|
|
318
|
+
takes an async storage adapter:
|
|
319
|
+
|
|
320
|
+
```python
|
|
321
|
+
from objbase.pydantic import AsyncPydanticInventory
|
|
322
|
+
|
|
323
|
+
todos = AsyncPydanticInventory(
|
|
324
|
+
item_type="todo",
|
|
325
|
+
storage=AsyncRedisInventoryStorage(redis.asyncio.Redis()),
|
|
326
|
+
model_class=Todo,
|
|
327
|
+
)
|
|
328
|
+
|
|
329
|
+
await todos.save(Todo(id="1", title="Buy milk"))
|
|
330
|
+
item = await todos.get("1") # Todo | None
|
|
331
|
+
```
|
|
332
|
+
|
|
333
|
+
---
|
|
334
|
+
|
|
335
|
+
## Async Usage
|
|
336
|
+
|
|
337
|
+
`AsyncInventory` has the same methods and behaviour as `Inventory`, but every
|
|
338
|
+
method is a coroutine. It works with any `AsyncInventoryStorage` adapter:
|
|
339
|
+
`AsyncFileBasedInventoryStorage`, `AsyncDirectoryBasedInventoryStorage`, `AsyncSQLiteInventoryStorage`,
|
|
340
|
+
`AsyncRedisInventoryStorage`, `AsyncMongoDBInventoryStorage`,
|
|
341
|
+
or `InMemoryInventoryStorage` for tests.
|
|
342
|
+
|
|
343
|
+
```python
|
|
344
|
+
import redis.asyncio
|
|
345
|
+
from objbase.asyncio.async_inventory import AsyncInventory
|
|
346
|
+
from objbase.asyncio.async_redis_storage import AsyncRedisInventoryStorage
|
|
347
|
+
|
|
348
|
+
client = redis.asyncio.Redis(host="localhost", port=6379)
|
|
349
|
+
todos = AsyncInventory(item_type="todo", storage=AsyncRedisInventoryStorage(client))
|
|
350
|
+
|
|
351
|
+
await todos.save({"id": "1", "title": "Buy milk", "done": False})
|
|
352
|
+
await todos.get("1") # → {"id": "1", "title": "Buy milk", "done": False}
|
|
353
|
+
await todos.filter() # → [{"id": "1", ...}]
|
|
354
|
+
await todos.keys() # → ["1"]
|
|
355
|
+
await todos.patch("1", {"done": True}) # → {"id": "1", ..., "done": True}
|
|
356
|
+
await todos.delete("1") # → True
|
|
357
|
+
```
|
|
358
|
+
|
|
359
|
+
For Pydantic models, use `AsyncPydanticInventory` (see [Pydantic Models: Async](#async)).
|
|
360
|
+
|
|
361
|
+
Passing a sync-only adapter (e.g. `SQLiteInventoryStorage`) to `AsyncInventory`
|
|
362
|
+
raises `TypeError`; use its async counterpart (e.g. `AsyncSQLiteInventoryStorage`) instead.
|
|
363
|
+
The adapter methods (`akeys`, `aitems`, `aread`, `awrite`, `adelete`)
|
|
364
|
+
can also be called directly on the storage.
|
|
365
|
+
|
|
366
|
+
---
|
|
367
|
+
|
|
368
|
+
## Examples
|
|
369
|
+
|
|
370
|
+
Runnable scripts are in [`examples/`](examples/):
|
|
371
|
+
|
|
372
|
+
| Script | Shows |
|
|
373
|
+
|---|---|
|
|
374
|
+
| `dict_example.py` | `Inventory` with plain dicts (in-memory) |
|
|
375
|
+
| `pydantic_example.py` | `PydanticInventory` (in-memory) |
|
|
376
|
+
| `async_example.py` | `AsyncInventory` (in-memory) |
|
|
377
|
+
| `async_pydantic_example.py` | `AsyncPydanticInventory` (in-memory) |
|
|
378
|
+
| `async_file_example.py` | `AsyncFileBasedInventoryStorage` |
|
|
379
|
+
| `async_directory_example.py` | `AsyncDirectoryBasedInventoryStorage`, incl. concurrent saves and `arebuild_index` |
|
|
380
|
+
| `async_sqlite_example.py` | `AsyncSQLiteInventoryStorage` |
|
|
381
|
+
| `mongodb_example.py` | `MongoDBInventoryStorage`, incl. a MongoDB `query` filter |
|
|
382
|
+
| `async_mongodb_example.py` | `AsyncMongoDBInventoryStorage`, incl. a MongoDB `query` filter |
|
|
383
|
+
|
|
384
|
+
```bash
|
|
385
|
+
uv run python examples/async_sqlite_example.py
|
|
386
|
+
```
|
|
387
|
+
|
|
388
|
+
The file-based and SQLite examples write to `data/` in the current directory
|
|
389
|
+
(ignored by git); set `INVENTORY_DIR` or `SQLITE_DB_PATH` to change that. The MongoDB
|
|
390
|
+
examples need a running server — `docker run --rm -p 27017:27017 mongo:7.0` — and
|
|
391
|
+
connect to `MONGODB_URI` (default `mongodb://localhost:27017`).
|
|
392
|
+
|
|
393
|
+
---
|
|
394
|
+
|
|
395
|
+
## FastAPI Integration
|
|
396
|
+
|
|
397
|
+
### 1. Initialise storage with `lifespan`
|
|
398
|
+
|
|
399
|
+
Use FastAPI's `lifespan` context manager to create the client and storage adapter
|
|
400
|
+
once at startup and tear them down cleanly on shutdown.
|
|
401
|
+
|
|
402
|
+
```python
|
|
403
|
+
from contextlib import asynccontextmanager
|
|
404
|
+
from fastapi import FastAPI
|
|
405
|
+
import redis.asyncio
|
|
406
|
+
from objbase.asyncio.async_redis_storage import AsyncRedisInventoryStorage
|
|
407
|
+
|
|
408
|
+
|
|
409
|
+
@asynccontextmanager
|
|
410
|
+
async def lifespan(app: FastAPI):
|
|
411
|
+
client = redis.asyncio.Redis(host="localhost", port=6379)
|
|
412
|
+
app.state.storage = AsyncRedisInventoryStorage(redis_client=client)
|
|
413
|
+
yield
|
|
414
|
+
await client.aclose()
|
|
415
|
+
|
|
416
|
+
|
|
417
|
+
app = FastAPI(lifespan=lifespan)
|
|
418
|
+
```
|
|
419
|
+
|
|
420
|
+
### 2. Inject `AsyncInventory` with `Depends`
|
|
421
|
+
|
|
422
|
+
Wrap the `AsyncInventory` construction in a dependency function so routes stay clean
|
|
423
|
+
and the storage adapter is easy to swap out (e.g. in tests).
|
|
424
|
+
|
|
425
|
+
```python
|
|
426
|
+
from fastapi import Depends, HTTPException, Request
|
|
427
|
+
from objbase.asyncio.async_inventory import AsyncInventory
|
|
428
|
+
from objbase.errors import ItemNotFoundError
|
|
429
|
+
|
|
430
|
+
|
|
431
|
+
def get_todos(request: Request) -> AsyncInventory:
|
|
432
|
+
return AsyncInventory(item_type="todo", storage=request.app.state.storage)
|
|
433
|
+
|
|
434
|
+
|
|
435
|
+
@app.get("/todos")
|
|
436
|
+
async def list_todos(todos: AsyncInventory = Depends(get_todos)):
|
|
437
|
+
return await todos.filter()
|
|
438
|
+
|
|
439
|
+
|
|
440
|
+
@app.get("/todos/{todo_id}")
|
|
441
|
+
async def get_todo(todo_id: str, todos: AsyncInventory = Depends(get_todos)):
|
|
442
|
+
item = await todos.get(todo_id)
|
|
443
|
+
if item is None:
|
|
444
|
+
raise HTTPException(status_code=404)
|
|
445
|
+
return item
|
|
446
|
+
|
|
447
|
+
|
|
448
|
+
@app.post("/todos")
|
|
449
|
+
async def create_todo(item: dict, todos: AsyncInventory = Depends(get_todos)):
|
|
450
|
+
return await todos.save(item)
|
|
451
|
+
|
|
452
|
+
|
|
453
|
+
@app.patch("/todos/{todo_id}")
|
|
454
|
+
async def update_todo(todo_id: str, data: dict, todos: AsyncInventory = Depends(get_todos)):
|
|
455
|
+
try:
|
|
456
|
+
return await todos.patch(todo_id, data)
|
|
457
|
+
except ItemNotFoundError:
|
|
458
|
+
raise HTTPException(status_code=404)
|
|
459
|
+
```
|
|
460
|
+
|
|
461
|
+
### 3. Sync routes with SQLite
|
|
462
|
+
|
|
463
|
+
For simpler apps without async requirements, SQLite is the easiest option.
|
|
464
|
+
Declare routes without `async def` — FastAPI runs them in a thread pool
|
|
465
|
+
automatically, keeping the event loop unblocked.
|
|
466
|
+
|
|
467
|
+
```python
|
|
468
|
+
from contextlib import asynccontextmanager
|
|
469
|
+
from fastapi import FastAPI, Depends, Request
|
|
470
|
+
from objbase.inventory import Inventory
|
|
471
|
+
from objbase.storage.sqlite_storage import SQLiteInventoryStorage
|
|
472
|
+
|
|
473
|
+
|
|
474
|
+
@asynccontextmanager
|
|
475
|
+
async def lifespan(app: FastAPI):
|
|
476
|
+
app.state.storage = SQLiteInventoryStorage("app.db")
|
|
477
|
+
yield
|
|
478
|
+
|
|
479
|
+
|
|
480
|
+
app = FastAPI(lifespan=lifespan)
|
|
481
|
+
|
|
482
|
+
|
|
483
|
+
def get_todos(request: Request) -> Inventory:
|
|
484
|
+
return Inventory(item_type="todo", storage=request.app.state.storage)
|
|
485
|
+
|
|
486
|
+
|
|
487
|
+
@app.get("/todos") # sync — runs in threadpool
|
|
488
|
+
def list_todos(todos: Inventory = Depends(get_todos)):
|
|
489
|
+
return todos.filter()
|
|
490
|
+
```
|
|
491
|
+
|
|
492
|
+
### 4. Override the dependency in tests
|
|
493
|
+
|
|
494
|
+
Swap the storage backend for the entire test run without touching any route code.
|
|
495
|
+
`InMemoryInventoryStorage` implements the async interface too, so it can stand in
|
|
496
|
+
for Redis. Create it once so data persists across requests:
|
|
497
|
+
|
|
498
|
+
```python
|
|
499
|
+
from objbase.asyncio.async_inventory import AsyncInventory
|
|
500
|
+
from objbase.storage.inmemory_storage import InMemoryInventoryStorage
|
|
501
|
+
from fastapi.testclient import TestClient
|
|
502
|
+
|
|
503
|
+
test_storage = InMemoryInventoryStorage()
|
|
504
|
+
|
|
505
|
+
|
|
506
|
+
def override_todos():
|
|
507
|
+
return AsyncInventory(item_type="todo", storage=test_storage)
|
|
508
|
+
|
|
509
|
+
|
|
510
|
+
app.dependency_overrides[get_todos] = override_todos
|
|
511
|
+
client = TestClient(app)
|
|
512
|
+
```
|
|
513
|
+
|
|
514
|
+
### Adapter recommendation by scenario
|
|
515
|
+
|
|
516
|
+
| Scenario | Recommended adapter |
|
|
517
|
+
|---|---|
|
|
518
|
+
| Single-process, low traffic | `SQLiteInventoryStorage` — zero deps, ACID, simple |
|
|
519
|
+
| Multi-worker / multi-process | `RedisInventoryStorage` or `MongoDBInventoryStorage` |
|
|
520
|
+
| Async routes | `AsyncInventory` + `AsyncRedisInventoryStorage` or `AsyncMongoDBInventoryStorage` — non-blocking, fits the event loop |
|
|
521
|
+
| Async routes, single process, no infrastructure | `AsyncInventory` + `AsyncSQLiteInventoryStorage` — zero deps, runs in a worker thread |
|
|
522
|
+
| Testing / local dev | `InMemoryInventoryStorage` — fast, no infrastructure needed |
|
|
523
|
+
|
|
524
|
+
---
|
|
525
|
+
|
|
526
|
+
## Writing a Custom Adapter
|
|
527
|
+
|
|
528
|
+
Both interfaces are defined as `typing.Protocol` with `@runtime_checkable`.
|
|
529
|
+
This means **no import or inheritance is required** — any class that implements
|
|
530
|
+
the right methods is automatically a valid adapter (structural subtyping).
|
|
531
|
+
|
|
532
|
+
### Sync — `InventoryStorage`
|
|
533
|
+
|
|
534
|
+
```python
|
|
535
|
+
# objbase/interface.py
|
|
536
|
+
from typing import Any, Protocol, runtime_checkable
|
|
537
|
+
|
|
538
|
+
Item = dict[str, Any]
|
|
539
|
+
|
|
540
|
+
|
|
541
|
+
@runtime_checkable
|
|
542
|
+
class InventoryStorage(Protocol):
|
|
543
|
+
def keys(self, item_type: str) -> list[str]: ...
|
|
544
|
+
def items(self, item_type: str) -> list[Item]: ...
|
|
545
|
+
def read(self, item_type: str, id: str) -> Item | None: ...
|
|
546
|
+
def write(self, item_type: str, item: Item) -> bool: ...
|
|
547
|
+
def delete(self, item_type: str, id: str) -> bool: ...
|
|
548
|
+
```
|
|
549
|
+
|
|
550
|
+
Every adapter must follow this contract (the shared test suite in
|
|
551
|
+
`tests/test_storage_contract.py` checks it):
|
|
552
|
+
|
|
553
|
+
- `keys` returns all item ids of a type as strings, or `[]` if there are none.
|
|
554
|
+
- `items` returns all items of a type, or `[]` if there are none.
|
|
555
|
+
- `read` returns the item, or `None` if it does not exist.
|
|
556
|
+
- `write` inserts the item, or replaces an existing item with the same id entirely.
|
|
557
|
+
- `delete` returns `True` if an item was removed, `False` if it did not exist.
|
|
558
|
+
- Returned items are independent copies; mutating them does not change stored data.
|
|
559
|
+
|
|
560
|
+
The order of `keys` and `items` is unspecified.
|
|
561
|
+
|
|
562
|
+
### Async — `AsyncInventoryStorage`
|
|
563
|
+
|
|
564
|
+
```python
|
|
565
|
+
# objbase/asyncio/async_storage.py
|
|
566
|
+
from typing import Protocol, runtime_checkable
|
|
567
|
+
from objbase.interface import Item
|
|
568
|
+
|
|
569
|
+
|
|
570
|
+
@runtime_checkable
|
|
571
|
+
class AsyncInventoryStorage(Protocol):
|
|
572
|
+
async def akeys(self, item_type: str) -> list[str]: ...
|
|
573
|
+
|
|
574
|
+
async def aitems(self, item_type: str) -> list[Item]: ...
|
|
575
|
+
|
|
576
|
+
async def aread(self, item_type: str, id: str) -> Item | None: ...
|
|
577
|
+
|
|
578
|
+
async def awrite(self, item_type: str, item: Item) -> bool: ...
|
|
579
|
+
|
|
580
|
+
async def adelete(self, item_type: str, id: str) -> bool: ...
|
|
581
|
+
```
|
|
582
|
+
|
|
583
|
+
Same contract as the sync protocol, with every method a coroutine.
|
|
584
|
+
|
|
585
|
+
### Duck-typing — no inheritance needed
|
|
586
|
+
|
|
587
|
+
Because `Protocol` uses structural subtyping, a third-party class is a valid
|
|
588
|
+
adapter as long as it has the right methods — it does not need to import or
|
|
589
|
+
subclass anything from `objbase`:
|
|
590
|
+
|
|
591
|
+
```python
|
|
592
|
+
class MyCustomStorage:
|
|
593
|
+
def keys(self, item_type: str) -> list[str]: ...
|
|
594
|
+
|
|
595
|
+
def items(self, item_type: str) -> list[dict]: ...
|
|
596
|
+
|
|
597
|
+
def read(self, item_type: str, id: str) -> dict | None: ...
|
|
598
|
+
|
|
599
|
+
def write(self, item_type: str, item: dict) -> bool: ...
|
|
600
|
+
|
|
601
|
+
def delete(self, item_type: str, id: str) -> bool: ...
|
|
602
|
+
|
|
603
|
+
|
|
604
|
+
# Works — no explicit inheritance required
|
|
605
|
+
todos = Inventory(item_type="todo", storage=MyCustomStorage())
|
|
606
|
+
```
|
|
607
|
+
|
|
608
|
+
### Optional explicit inheritance
|
|
609
|
+
|
|
610
|
+
You can still inherit from the Protocol if you want IDE support for "find all
|
|
611
|
+
implementations" or early feedback from a type checker when a method is missing:
|
|
612
|
+
|
|
613
|
+
```python
|
|
614
|
+
from objbase.interface import InventoryStorage
|
|
615
|
+
|
|
616
|
+
|
|
617
|
+
class MyCustomStorage(InventoryStorage): # explicit, but optional
|
|
618
|
+
...
|
|
619
|
+
```
|
|
620
|
+
|
|
621
|
+
### Runtime checks with `isinstance`
|
|
622
|
+
|
|
623
|
+
Both protocols are `@runtime_checkable`, so you can verify at runtime that an
|
|
624
|
+
object has the required methods. This checks method names only, not signatures;
|
|
625
|
+
use a type checker for full verification:
|
|
626
|
+
|
|
627
|
+
```python
|
|
628
|
+
from objbase.interface import InventoryStorage
|
|
629
|
+
|
|
630
|
+
isinstance(MyCustomStorage(), InventoryStorage) # True
|
|
631
|
+
isinstance("not a storage", InventoryStorage) # False
|
|
632
|
+
```
|
|
633
|
+
|
|
634
|
+
---
|
|
635
|
+
|
|
636
|
+
## Type Hints
|
|
637
|
+
|
|
638
|
+
The package ships a `py.typed` marker, so mypy, Pyright and IDEs use its type
|
|
639
|
+
hints. The library itself is checked with `mypy --strict`.
|
|
640
|
+
|
|
641
|
+
- Items are typed as `objbase.Item`, an alias for `dict[str, Any]`.
|
|
642
|
+
- `Inventory` and `AsyncInventory` accept and return `Item`; `get` returns `Item | None`.
|
|
643
|
+
- `PydanticInventory` and `AsyncPydanticInventory` are generic over their model class, which is inferred from
|
|
644
|
+
`model_class` (see [Pydantic Models](#pydantic-models)).
|
|
645
|
+
- Storage adapters accept any structurally compatible client. For example,
|
|
646
|
+
`RedisInventoryStorage` takes anything with Redis's `hget`/`hset`/`hdel`/`hvals`
|
|
647
|
+
commands (`redis.Redis`, `redis.asyncio.Redis`, or compatible clients).
|
|
648
|
+
|
|
649
|
+
---
|
|
650
|
+
|
|
651
|
+
## Development
|
|
652
|
+
|
|
653
|
+
Requires [uv](https://docs.astral.sh/uv/). Install the package with all
|
|
654
|
+
development dependencies (pinned in `uv.lock`):
|
|
655
|
+
|
|
656
|
+
```bash
|
|
657
|
+
uv sync
|
|
658
|
+
```
|
|
659
|
+
|
|
660
|
+
### Tests
|
|
661
|
+
|
|
662
|
+
```bash
|
|
663
|
+
uv run pytest
|
|
664
|
+
```
|
|
665
|
+
|
|
666
|
+
The Redis and MongoDB tests start containers via
|
|
667
|
+
[testcontainers](https://testcontainers.com/), so Docker must be running. Without
|
|
668
|
+
Docker, the shared contract tests skip those backends, but the Redis and MongoDB test
|
|
669
|
+
modules fail; exclude them to run everything else:
|
|
670
|
+
|
|
671
|
+
```bash
|
|
672
|
+
uv run pytest --ignore=tests/test_redis_storage.py --ignore=tests/test_async_redis_storage.py \
|
|
673
|
+
--ignore=tests/test_mongodb_storage.py --ignore=tests/test_async_mongodb_storage.py
|
|
674
|
+
```
|
|
675
|
+
|
|
676
|
+
MongoDB tests use
|
|
677
|
+
`mongo:7.0`, because `mongo:latest` does not start on Linux kernels 6.19+ (as used by
|
|
678
|
+
recent Docker Desktop VMs). Override the image with `INVENTORYDB_TEST_MONGO_IMAGE`.
|
|
679
|
+
|
|
680
|
+
### Linting and formatting
|
|
681
|
+
|
|
682
|
+
[Ruff](https://docs.astral.sh/ruff/) checks for likely bugs, style issues, import
|
|
683
|
+
order and outdated syntax. The enabled rules are listed under `[tool.ruff.lint]` in
|
|
684
|
+
`pyproject.toml`.
|
|
685
|
+
|
|
686
|
+
```bash
|
|
687
|
+
uv run ruff check . # report issues
|
|
688
|
+
uv run ruff check --fix . # apply safe automatic fixes
|
|
689
|
+
```
|
|
690
|
+
|
|
691
|
+
Code is formatted with Ruff's formatter (line length 120, set under `[tool.ruff]`).
|
|
692
|
+
CI fails if any file is not formatted:
|
|
693
|
+
|
|
694
|
+
```bash
|
|
695
|
+
uv run ruff format . # format all files
|
|
696
|
+
uv run ruff format --check . # check only, as CI does
|
|
697
|
+
```
|
|
698
|
+
|
|
699
|
+
### Type checking
|
|
700
|
+
|
|
701
|
+
[mypy](https://mypy.readthedocs.io/) checks the library in strict mode (configured
|
|
702
|
+
under `[tool.mypy]` in `pyproject.toml`):
|
|
703
|
+
|
|
704
|
+
```bash
|
|
705
|
+
uv run mypy
|
|
706
|
+
```
|
|
707
|
+
|
|
708
|
+
Tests and examples are checked too, with rules for unannotated test functions
|
|
709
|
+
relaxed. They use the public API the way users do, so this catches annotations
|
|
710
|
+
that are correct internally but awkward for callers:
|
|
711
|
+
|
|
712
|
+
```bash
|
|
713
|
+
uv run mypy --allow-untyped-defs --allow-incomplete-defs --allow-untyped-calls tests examples
|
|
714
|
+
```
|
|
715
|
+
|
|
716
|
+
### Continuous integration
|
|
717
|
+
|
|
718
|
+
[GitHub Actions](.github/workflows/ci.yml) runs on every push to `main`, every
|
|
719
|
+
pull request, and as the first stage of every [release](#releasing):
|
|
720
|
+
|
|
721
|
+
| Job | What it does |
|
|
722
|
+
|---|---|
|
|
723
|
+
| Lint, format and type check | Ruff lint, Ruff format check, and the mypy commands above (the library is also checked as Windows sees it, with `--platform win32`) |
|
|
724
|
+
| Test | Full test suite on Python 3.13 and 3.14 |
|
|
725
|
+
| Test (Windows / macOS, no containers) | Test suite without the Redis and MongoDB tests, covering platform-specific code such as file locking |
|
|
726
|
+
| Test (minimum dependency versions) | Test suite with the lowest versions of `redis`, `pymongo` and `pydantic` allowed by `pyproject.toml` |
|
|
727
|
+
| Build distributions | Builds the sdist and wheel, and checks their metadata and contents |
|
|
728
|
+
|
|
729
|
+
Run the lint, format, type check and test commands above before pushing to catch
|
|
730
|
+
failures early.
|
|
731
|
+
|
|
732
|
+
[Dependabot](.github/dependabot.yml) checks weekly for updates and skips
|
|
733
|
+
releases less than a week old:
|
|
734
|
+
|
|
735
|
+
- **GitHub Actions:** all actions in both workflows are pinned to commit SHAs;
|
|
736
|
+
one PR updates the SHAs and their version comments.
|
|
737
|
+
- **Python dependencies:** PRs that update `uv.lock`, one for the backend
|
|
738
|
+
libraries (`redis`, `pymongo`, `pydantic`) and one for dev tools. The `>=`
|
|
739
|
+
minimum versions in `pyproject.toml` are left unchanged.
|
|
740
|
+
|
|
741
|
+
### Releasing
|
|
742
|
+
|
|
743
|
+
Releases are published by the [release workflow](.github/workflows/release.yml)
|
|
744
|
+
when a tag starting with `v` is pushed. Bump the version, commit, then tag the
|
|
745
|
+
commit with the same version:
|
|
746
|
+
|
|
747
|
+
```bash
|
|
748
|
+
uv version 0.3.0
|
|
749
|
+
git commit -am "release 0.3.0"
|
|
750
|
+
git tag v0.3.0
|
|
751
|
+
git push origin main v0.3.0
|
|
752
|
+
```
|
|
753
|
+
|
|
754
|
+
The workflow then:
|
|
755
|
+
|
|
756
|
+
1. Checks that the tag matches the version in `pyproject.toml` (`v0.3.0` ↔ `0.3.0`) and fails otherwise.
|
|
757
|
+
2. Runs the full CI workflow.
|
|
758
|
+
3. Builds the sdist and wheel and checks their metadata.
|
|
759
|
+
4. Publishes to TestPyPI and checks that the new version installs from there.
|
|
760
|
+
If either fails, nothing is published to PyPI.
|
|
761
|
+
5. Publishes to PyPI. Both uploads use
|
|
762
|
+
[trusted publishing](https://docs.pypi.org/trusted-publishers/), so no API tokens are stored.
|
|
763
|
+
6. Creates a GitHub release for the tag with generated release notes and the
|
|
764
|
+
distributions attached. Pre-release versions (`a`, `b`, `rc`, `.dev`) are
|
|
765
|
+
marked as pre-releases.
|
|
766
|
+
|
|
767
|
+
One-time setup:
|
|
768
|
+
|
|
769
|
+
- On [PyPI](https://pypi.org), add a trusted publisher for the `fm-labs/objbase`
|
|
770
|
+
repository with workflow `release.yml` and environment `pypi`.
|
|
771
|
+
- On [TestPyPI](https://test.pypi.org), add the same trusted publisher with
|
|
772
|
+
environment `testpypi`.
|
|
773
|
+
- In the GitHub repository settings, create the `testpypi` and `pypi`
|
|
774
|
+
environments. Add required reviewers to `pypi` to approve each release after
|
|
775
|
+
the TestPyPI check and before it's published.
|
|
776
|
+
|
|
777
|
+
To publish from a local machine instead, `release.sh` refuses to run with
|
|
778
|
+
uncommitted changes, runs the tests, builds into a clean `dist/`, and publishes
|
|
779
|
+
to TestPyPI and/or PyPI depending on which of `TESTPYPI_PUBLISH_TOKEN` and
|
|
780
|
+
`PYPI_PUBLISH_TOKEN` are set.
|