opendataframework 0.0.4__tar.gz → 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {opendataframework-0.0.4 → opendataframework-0.2.0}/LICENSE +1 -1
- opendataframework-0.2.0/PKG-INFO +356 -0
- opendataframework-0.2.0/README.md +339 -0
- opendataframework-0.2.0/pyproject.toml +65 -0
- opendataframework-0.2.0/src/opendataframework/__init__.py +74 -0
- opendataframework-0.2.0/src/opendataframework/component.py +177 -0
- opendataframework-0.2.0/src/opendataframework/config.py +163 -0
- opendataframework-0.2.0/src/opendataframework/context.py +885 -0
- opendataframework-0.2.0/src/opendataframework/entity.py +52 -0
- opendataframework-0.2.0/src/opendataframework/layer.py +130 -0
- opendataframework-0.2.0/src/opendataframework/logger.py +139 -0
- opendataframework-0.2.0/src/opendataframework/namespace.py +189 -0
- opendataframework-0.2.0/src/opendataframework/pipeline.py +88 -0
- opendataframework-0.2.0/src/opendataframework/project.py +180 -0
- opendataframework-0.2.0/src/opendataframework/repository.py +210 -0
- opendataframework-0.2.0/src/opendataframework/service.py +89 -0
- opendataframework-0.2.0/src/opendataframework/task.py +68 -0
- opendataframework-0.2.0/src/opendataframework/utils.py +26 -0
- opendataframework-0.2.0/src/opendataframework/view.py +189 -0
- opendataframework-0.0.4/PKG-INFO +0 -19
- opendataframework-0.0.4/README.md +0 -2
- opendataframework-0.0.4/pyproject.toml +0 -48
- opendataframework-0.0.4/setup.py +0 -73
- opendataframework-0.0.4/src/opendataframework/README.md +0 -43
- opendataframework-0.0.4/src/opendataframework/__init__.py +0 -3
- opendataframework-0.0.4/src/opendataframework/__main__.py +0 -2143
- opendataframework-0.0.4/src/opendataframework/analytics/setup.sh +0 -1
- opendataframework-0.0.4/src/opendataframework/analytics/superset/.env +0 -1
- opendataframework-0.0.4/src/opendataframework/analytics/superset/Dockerfile +0 -3
- opendataframework-0.0.4/src/opendataframework/analytics/superset/database/postgres/create.sh +0 -10
- opendataframework-0.0.4/src/opendataframework/analytics/superset/database/postgres/setup.sh +0 -25
- opendataframework-0.0.4/src/opendataframework/analytics/superset/docker-compose.yaml +0 -15
- opendataframework-0.0.4/src/opendataframework/analytics/superset/setup.sh +0 -16
- opendataframework-0.0.4/src/opendataframework/api/api-postgres/.env +0 -9
- opendataframework-0.0.4/src/opendataframework/api/api-postgres/Dockerfile +0 -16
- opendataframework-0.0.4/src/opendataframework/api/api-postgres/app/__init__.py +0 -11
- opendataframework-0.0.4/src/opendataframework/api/api-postgres/app/config.py +0 -17
- opendataframework-0.0.4/src/opendataframework/api/api-postgres/app/crud.py +0 -31
- opendataframework-0.0.4/src/opendataframework/api/api-postgres/app/database.py +0 -12
- opendataframework-0.0.4/src/opendataframework/api/api-postgres/app/dependencies.py +0 -11
- opendataframework-0.0.4/src/opendataframework/api/api-postgres/app/main.py +0 -46
- opendataframework-0.0.4/src/opendataframework/api/api-postgres/app/models.py +0 -32
- opendataframework-0.0.4/src/opendataframework/api/api-postgres/app/router.py +0 -76
- opendataframework-0.0.4/src/opendataframework/api/api-postgres/docker-compose.yaml +0 -16
- opendataframework-0.0.4/src/opendataframework/api/api-postgres/poetry.lock +0 -486
- opendataframework-0.0.4/src/opendataframework/api/api-postgres/pyproject.toml +0 -20
- opendataframework-0.0.4/src/opendataframework/api/inference/.env +0 -7
- opendataframework-0.0.4/src/opendataframework/api/inference/Dockerfile +0 -16
- opendataframework-0.0.4/src/opendataframework/api/inference/app/__init__.py +0 -11
- opendataframework-0.0.4/src/opendataframework/api/inference/app/config.py +0 -14
- opendataframework-0.0.4/src/opendataframework/api/inference/app/crud.py +0 -14
- opendataframework-0.0.4/src/opendataframework/api/inference/app/dependencies.py +0 -19
- opendataframework-0.0.4/src/opendataframework/api/inference/app/main.py +0 -39
- opendataframework-0.0.4/src/opendataframework/api/inference/app/models.py +0 -20
- opendataframework-0.0.4/src/opendataframework/api/inference/app/router.py +0 -18
- opendataframework-0.0.4/src/opendataframework/api/inference/docker-compose.yaml +0 -16
- opendataframework-0.0.4/src/opendataframework/api/inference/pyproject.toml +0 -20
- opendataframework-0.0.4/src/opendataframework/api/ingest.py +0 -179
- opendataframework-0.0.4/src/opendataframework/api/requirements.txt +0 -1
- opendataframework-0.0.4/src/opendataframework/api/setup.sh +0 -1
- opendataframework-0.0.4/src/opendataframework/devcontainers/R/.devcontainer/.Rprofile +0 -17
- opendataframework-0.0.4/src/opendataframework/devcontainers/R/.devcontainer/Dockerfile +0 -34
- opendataframework-0.0.4/src/opendataframework/devcontainers/R/.devcontainer/devcontainer.json +0 -43
- opendataframework-0.0.4/src/opendataframework/devcontainers/python/.devcontainer/Dockerfile +0 -3
- opendataframework-0.0.4/src/opendataframework/devcontainers/python/.devcontainer/devcontainer.json +0 -23
- opendataframework-0.0.4/src/opendataframework/docker-compose.yaml +0 -2
- opendataframework-0.0.4/src/opendataframework/docs/index.md +0 -1
- opendataframework-0.0.4/src/opendataframework/docs/mkdocs.yml +0 -56
- opendataframework-0.0.4/src/opendataframework/docs/requirements.txt +0 -1
- opendataframework-0.0.4/src/opendataframework/docs/stylesheets/extra.css +0 -8
- opendataframework-0.0.4/src/opendataframework/expectations.py +0 -76
- opendataframework-0.0.4/src/opendataframework/github/workflows/ci.yaml +0 -21
- opendataframework-0.0.4/src/opendataframework/hooks/.pre-commit-config.yaml +0 -29
- opendataframework-0.0.4/src/opendataframework/hooks/requirements.txt +0 -1
- opendataframework-0.0.4/src/opendataframework/layouts/research/.vscode/extensions.json +0 -8
- opendataframework-0.0.4/src/opendataframework/layouts/research/code/check/.gitkeep +0 -0
- opendataframework-0.0.4/src/opendataframework/layouts/research/code/learn/.gitkeep +0 -0
- opendataframework-0.0.4/src/opendataframework/layouts/research/code/share/.gitkeep +0 -0
- opendataframework-0.0.4/src/opendataframework/layouts/research/data/derived/.gitkeep +0 -0
- opendataframework-0.0.4/src/opendataframework/layouts/research/data/raw/.gitkeep +0 -0
- opendataframework-0.0.4/src/opendataframework/layouts/research/libraries/.gitkeep +0 -0
- opendataframework-0.0.4/src/opendataframework/layouts/research/logs/.gitkeep +0 -0
- opendataframework-0.0.4/src/opendataframework/layouts/research/models/.gitkeep +0 -0
- opendataframework-0.0.4/src/opendataframework/layouts/research/output/figures/.gitkeep +0 -0
- opendataframework-0.0.4/src/opendataframework/layouts/research/output/tables/.gitkeep +0 -0
- opendataframework-0.0.4/src/opendataframework/layouts/research/paper/.gitkeep +0 -0
- opendataframework-0.0.4/src/opendataframework/requirements.txt +0 -1
- opendataframework-0.0.4/src/opendataframework/setup.sh +0 -3
- opendataframework-0.0.4/src/opendataframework/storage/postgres/.env +0 -1
- opendataframework-0.0.4/src/opendataframework/storage/postgres/Dockerfile +0 -1
- opendataframework-0.0.4/src/opendataframework/storage/postgres/docker-compose.yaml +0 -16
- opendataframework-0.0.4/src/opendataframework/storage/postgres/setup.sh +0 -5
- opendataframework-0.0.4/src/opendataframework/storage/setup.sh +0 -1
- opendataframework-0.0.4/src/opendataframework/tests/__init__.py +0 -1
- opendataframework-0.0.4/src/opendataframework/tests/requirements.txt +0 -2
- opendataframework-0.0.4/src/opendataframework/tests/test_example.py +0 -6
- opendataframework-0.0.4/src/opendataframework/utility/nginx/Dockerfile +0 -5
- opendataframework-0.0.4/src/opendataframework/utility/nginx/docker-compose.yaml +0 -14
- opendataframework-0.0.4/src/opendataframework/utility/nginx/nginx.conf +0 -6
- opendataframework-0.0.4/src/opendataframework/utility/nginx/static/images/logo.svg +0 -7
- opendataframework-0.0.4/src/opendataframework/utility/nginx/static/index.html +0 -39
- opendataframework-0.0.4/src/opendataframework/utility/setup.sh +0 -1
- opendataframework-0.0.4/src/opendataframework/utility/texlive/Dockerfile +0 -3
- opendataframework-0.0.4/src/opendataframework/utility/texlive/docker-compose.yaml +0 -15
- opendataframework-0.0.4/src/opendataframework/utility/texlive/mnt/.gitkeep +0 -0
- /opendataframework-0.0.4/src/opendataframework/layouts/research/code/build/.gitkeep → /opendataframework-0.2.0/src/opendataframework/py.typed +0 -0
|
@@ -0,0 +1,356 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: opendataframework
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Lightweight, config-driven dependency-injection framework for data applications.
|
|
5
|
+
License-Expression: MIT
|
|
6
|
+
License-File: LICENSE
|
|
7
|
+
Author: mykytapavlov
|
|
8
|
+
Author-email: mykytapavlov@users.noreply.github.com
|
|
9
|
+
Requires-Python: >=3.14
|
|
10
|
+
Classifier: Programming Language :: Python :: 3
|
|
11
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
12
|
+
Project-URL: Documentation, https://opendataframework.github.io/opendataframework/
|
|
13
|
+
Project-URL: Homepage, https://opendataframework.github.io/opendataframework/
|
|
14
|
+
Project-URL: Repository, https://github.com/opendataframework/opendataframework
|
|
15
|
+
Description-Content-Type: text/markdown
|
|
16
|
+
|
|
17
|
+
# Open Data Framework
|
|
18
|
+
|
|
19
|
+
[](https://pypi.org/project/opendataframework/)
|
|
20
|
+
[](https://www.python.org/downloads/)
|
|
21
|
+
[](LICENSE)
|
|
22
|
+
[](https://opendataframework.github.io/opendataframework/)
|
|
23
|
+
|
|
24
|
+
Lightweight dependency-injection framework for data applications.
|
|
25
|
+
|
|
26
|
+
This package provides structural abstractions —
|
|
27
|
+
`Entity`, `Repository`, `Component`, `Service`, `Task`, `Pipeline`,
|
|
28
|
+
`Layer`, `Namespace`, `Context`, `Project`, `Config`, `Logger`, `View`
|
|
29
|
+
— with zero third-party dependencies. No UI, no CLI, no MCP server.
|
|
30
|
+
|
|
31
|
+
> Need the CLI, UI, or MCP server? That's the
|
|
32
|
+
> [`odf`](https://opendataframework.github.io/odf/) package, which
|
|
33
|
+
> depends on this one.
|
|
34
|
+
|
|
35
|
+
Full documentation: **[opendataframework.github.io/opendataframework](https://opendataframework.github.io/opendataframework/)**
|
|
36
|
+
|
|
37
|
+
---
|
|
38
|
+
|
|
39
|
+
# Why
|
|
40
|
+
|
|
41
|
+
Most data projects solve the same structural problems repeatedly: how to
|
|
42
|
+
wire dependencies, manage component lifecycles, and keep configuration
|
|
43
|
+
separate from code. This framework provides that plumbing as a small set
|
|
44
|
+
of reusable abstractions — `Entity`, `Repository`, `Component`, `Service`,
|
|
45
|
+
`Task`, `Pipeline` — so it doesn't need to be reinvented by hand in every
|
|
46
|
+
project.
|
|
47
|
+
|
|
48
|
+
> Want a ready-made project layout instead? Scaffolding an actual project
|
|
49
|
+
> from a template is the [`odf`](https://opendataframework.github.io/odf/)
|
|
50
|
+
> package's job — this is what that scaffold is built on.
|
|
51
|
+
|
|
52
|
+
---
|
|
53
|
+
|
|
54
|
+
# Getting Started
|
|
55
|
+
|
|
56
|
+
Install:
|
|
57
|
+
|
|
58
|
+
```bash
|
|
59
|
+
pip install opendataframework
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
A minimal project needs three things: an entity and repository, a config
|
|
63
|
+
file, and a main file that starts the `Project`.
|
|
64
|
+
|
|
65
|
+
Application code (`app.py`):
|
|
66
|
+
|
|
67
|
+
```python
|
|
68
|
+
import sqlite3
|
|
69
|
+
from dataclasses import dataclass
|
|
70
|
+
|
|
71
|
+
from opendataframework import Component, Config, Entity, Repository
|
|
72
|
+
|
|
73
|
+
|
|
74
|
+
@Entity
|
|
75
|
+
@dataclass
|
|
76
|
+
class User:
|
|
77
|
+
id: int
|
|
78
|
+
name: str
|
|
79
|
+
|
|
80
|
+
|
|
81
|
+
@Component
|
|
82
|
+
class SQLite:
|
|
83
|
+
def __init__(self, config: Config):
|
|
84
|
+
self.conn = sqlite3.connect(config.sqlite.path)
|
|
85
|
+
self.conn.execute("CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT)")
|
|
86
|
+
|
|
87
|
+
|
|
88
|
+
@Repository(User)
|
|
89
|
+
class Users:
|
|
90
|
+
def __init__(self, sqlite: SQLite):
|
|
91
|
+
self.db = sqlite.conn
|
|
92
|
+
|
|
93
|
+
def all(self) -> list[User]:
|
|
94
|
+
rows = self.db.execute("SELECT id, name FROM users").fetchall()
|
|
95
|
+
return [User(id=row[0], name=row[1]) for row in rows]
|
|
96
|
+
|
|
97
|
+
def save(self, user: User) -> None:
|
|
98
|
+
self.db.execute("INSERT OR REPLACE INTO users VALUES (?, ?)", (user.id, user.name))
|
|
99
|
+
self.db.commit()
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
Config file (`config.toml`):
|
|
103
|
+
|
|
104
|
+
```toml
|
|
105
|
+
[sqlite]
|
|
106
|
+
path = "app.db"
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
Main file (`main.py`):
|
|
110
|
+
|
|
111
|
+
```python
|
|
112
|
+
from opendataframework import Project
|
|
113
|
+
|
|
114
|
+
from app import User, Users
|
|
115
|
+
|
|
116
|
+
project = Project.from_config("config.toml")
|
|
117
|
+
project.start()
|
|
118
|
+
|
|
119
|
+
users = project.context.get(Users)
|
|
120
|
+
users.save(User(id=1, name="Ada"))
|
|
121
|
+
|
|
122
|
+
print(users.all())
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
`project.start()` blocks until every component has completed its
|
|
126
|
+
initialisation stage — there is no hidden latency on first access. Adding
|
|
127
|
+
a new component means declaring its dependencies in the constructor, not
|
|
128
|
+
writing wiring code — a config section is only needed if the component
|
|
129
|
+
reads from `Config` itself.
|
|
130
|
+
|
|
131
|
+
---
|
|
132
|
+
|
|
133
|
+
# Core Concepts
|
|
134
|
+
|
|
135
|
+
## Entity
|
|
136
|
+
|
|
137
|
+
A structured unit of data the system works with.
|
|
138
|
+
|
|
139
|
+
```python
|
|
140
|
+
@Entity
|
|
141
|
+
@dataclass
|
|
142
|
+
class User:
|
|
143
|
+
id: int
|
|
144
|
+
name: str
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
## Repository
|
|
148
|
+
|
|
149
|
+
Manages Entities; abstracts persistence away from business logic.
|
|
150
|
+
|
|
151
|
+
```python
|
|
152
|
+
@Repository(User)
|
|
153
|
+
class Users:
|
|
154
|
+
def get(self, user_id: int) -> User: ...
|
|
155
|
+
def save(self, user: User) -> None: ...
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
## View
|
|
159
|
+
|
|
160
|
+
An optional, single declaration a `Repository` attaches to itself: which
|
|
161
|
+
representation (table, map, image, video, ...) and field(s) best fit its
|
|
162
|
+
data. Metadata only — this package renders nothing itself. See
|
|
163
|
+
[View](https://opendataframework.github.io/opendataframework/view/).
|
|
164
|
+
|
|
165
|
+
```python
|
|
166
|
+
@Repository(Store)
|
|
167
|
+
class Stores:
|
|
168
|
+
def all(self) -> list[Store]: ...
|
|
169
|
+
|
|
170
|
+
def data_view(self) -> LocationView:
|
|
171
|
+
return LocationView(fields=("lat", "lon"))
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
## Component
|
|
175
|
+
|
|
176
|
+
A `Context`-managed object with no required execution contract — wired in
|
|
177
|
+
and given its dependencies through the constructor.
|
|
178
|
+
|
|
179
|
+
```python
|
|
180
|
+
@Component
|
|
181
|
+
class Classifier:
|
|
182
|
+
def fit(self, data): ...
|
|
183
|
+
def predict(self, data): ...
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
## Service
|
|
187
|
+
|
|
188
|
+
A long-running executable, wired through DI the same way as a `Component` but
|
|
189
|
+
with a mandatory `setup → run → stop` lifecycle — an API server, worker
|
|
190
|
+
process, or scheduler. `run()` blocks by nature; the framework backgrounds it
|
|
191
|
+
automatically.
|
|
192
|
+
|
|
193
|
+
```python
|
|
194
|
+
@Service
|
|
195
|
+
class Postgres:
|
|
196
|
+
def setup(self): ...
|
|
197
|
+
def run(self): ...
|
|
198
|
+
def stop(self): ...
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
## Task
|
|
202
|
+
|
|
203
|
+
A finite executable, wired through DI the same way as a `Component` —
|
|
204
|
+
performs one bounded unit of work, once.
|
|
205
|
+
|
|
206
|
+
```python
|
|
207
|
+
@Task
|
|
208
|
+
class MetricsFetcher:
|
|
209
|
+
def execute(self): ...
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
## Pipeline
|
|
213
|
+
|
|
214
|
+
Coordinates multiple `Task`s (and optionally other `Pipeline`s) into an
|
|
215
|
+
ordered workflow.
|
|
216
|
+
|
|
217
|
+
```python
|
|
218
|
+
@Pipeline
|
|
219
|
+
class DailyAnalytics:
|
|
220
|
+
def execute(self): ...
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
## Layer
|
|
224
|
+
|
|
225
|
+
An organisational grouping — which subsystem a component belongs to
|
|
226
|
+
(`Api`, `Storage`, `Analytics`, `Messaging`, `Monitoring`, `Security`, or
|
|
227
|
+
custom). Independent of execution type.
|
|
228
|
+
|
|
229
|
+
```python
|
|
230
|
+
@Storage
|
|
231
|
+
@Service
|
|
232
|
+
class Postgres: ...
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
## Namespace
|
|
236
|
+
|
|
237
|
+
The base class underlying every framework decorator — `Entity`,
|
|
238
|
+
`Component`, `Service`, `Task`, `Pipeline`, `Repository`, and `Layer` are
|
|
239
|
+
all `Namespace` subclasses, each with its own independent name → class
|
|
240
|
+
mapping. A framework extension point, not something end users subclass
|
|
241
|
+
directly. See
|
|
242
|
+
[Namespace](https://opendataframework.github.io/opendataframework/namespace/).
|
|
243
|
+
|
|
244
|
+
## Config
|
|
245
|
+
|
|
246
|
+
Typed, dot-accessible view over a configuration dict — snake_case and
|
|
247
|
+
kebab-case are interchangeable, and nested sections chain naturally.
|
|
248
|
+
Resolved automatically into the container by `Project.from_config`.
|
|
249
|
+
|
|
250
|
+
```python
|
|
251
|
+
@Storage
|
|
252
|
+
@Service
|
|
253
|
+
class Postgres:
|
|
254
|
+
def __init__(self, config: Config):
|
|
255
|
+
self.host = config.postgres.host
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
## Logger
|
|
259
|
+
|
|
260
|
+
Per-component logging handle, injectable via constructor DI like any
|
|
261
|
+
other dependency — writes land in that component's own log file with no
|
|
262
|
+
name argument needed.
|
|
263
|
+
|
|
264
|
+
```python
|
|
265
|
+
@Task
|
|
266
|
+
class ExportUsers:
|
|
267
|
+
def __init__(self, users: Users, logger: Logger):
|
|
268
|
+
self.users = users
|
|
269
|
+
self.logger = logger
|
|
270
|
+
|
|
271
|
+
def execute(self) -> None:
|
|
272
|
+
self.logger.info("export starting")
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
## Context
|
|
276
|
+
|
|
277
|
+
Registers, resolves, and lifecycles every component in dependency order.
|
|
278
|
+
No circular dependencies — restructure with a shared third collaborator
|
|
279
|
+
instead.
|
|
280
|
+
|
|
281
|
+
```python
|
|
282
|
+
component = project.context.get(UsersApi) # typed, by class
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
## Project
|
|
286
|
+
|
|
287
|
+
The composition root — owns `Context` and configuration, and is the
|
|
288
|
+
single entry point for starting and stopping the application.
|
|
289
|
+
|
|
290
|
+
```python
|
|
291
|
+
project = Project.from_config("config.toml")
|
|
292
|
+
project.start()
|
|
293
|
+
```
|
|
294
|
+
|
|
295
|
+
Multiple independent `Project` instances are supported in the same
|
|
296
|
+
process; there is no global mutable state.
|
|
297
|
+
|
|
298
|
+
---
|
|
299
|
+
|
|
300
|
+
# Design Guidelines
|
|
301
|
+
|
|
302
|
+
* **Prefer composition** — decorators (`@Api @Service`), not inheritance
|
|
303
|
+
from framework base classes.
|
|
304
|
+
* **Keep responsibilities focused** — one primary reason to change per
|
|
305
|
+
class.
|
|
306
|
+
* **Avoid global mutable state** — framework state belongs to objects.
|
|
307
|
+
* **Avoid hidden magic** — dependencies must be visible in constructor
|
|
308
|
+
signatures.
|
|
309
|
+
|
|
310
|
+
---
|
|
311
|
+
|
|
312
|
+
# Mental Model
|
|
313
|
+
|
|
314
|
+
```text
|
|
315
|
+
Data is represented by Entities.
|
|
316
|
+
Repositories manage data.
|
|
317
|
+
|
|
318
|
+
Components provide capabilities.
|
|
319
|
+
Services run for extended periods.
|
|
320
|
+
Tasks execute bounded work.
|
|
321
|
+
Pipelines coordinate tasks (and other pipelines).
|
|
322
|
+
|
|
323
|
+
Layers organize components.
|
|
324
|
+
Context manages components and repositories.
|
|
325
|
+
Project owns everything.
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
---
|
|
329
|
+
|
|
330
|
+
# Examples
|
|
331
|
+
|
|
332
|
+
See [`examples/`](examples/README.md) — small, focused projects each
|
|
333
|
+
isolating one core abstraction.
|
|
334
|
+
|
|
335
|
+
> For the CLI, UI, MCP server, and chat surface
|
|
336
|
+
> built on top of this package, see `odf`.
|
|
337
|
+
|
|
338
|
+
---
|
|
339
|
+
|
|
340
|
+
# Development
|
|
341
|
+
|
|
342
|
+
Requires Python >=3.14, managed with [Poetry](https://python-poetry.org/).
|
|
343
|
+
|
|
344
|
+
```bash
|
|
345
|
+
poetry install # install dependencies, including dev
|
|
346
|
+
poetry run pytest # run the test suite
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
See [`CONTRIBUTING.md`](CONTRIBUTING.md) for how to propose changes.
|
|
350
|
+
|
|
351
|
+
---
|
|
352
|
+
|
|
353
|
+
# License
|
|
354
|
+
|
|
355
|
+
[MIT](LICENSE)
|
|
356
|
+
|
|
@@ -0,0 +1,339 @@
|
|
|
1
|
+
# Open Data Framework
|
|
2
|
+
|
|
3
|
+
[](https://pypi.org/project/opendataframework/)
|
|
4
|
+
[](https://www.python.org/downloads/)
|
|
5
|
+
[](LICENSE)
|
|
6
|
+
[](https://opendataframework.github.io/opendataframework/)
|
|
7
|
+
|
|
8
|
+
Lightweight dependency-injection framework for data applications.
|
|
9
|
+
|
|
10
|
+
This package provides structural abstractions —
|
|
11
|
+
`Entity`, `Repository`, `Component`, `Service`, `Task`, `Pipeline`,
|
|
12
|
+
`Layer`, `Namespace`, `Context`, `Project`, `Config`, `Logger`, `View`
|
|
13
|
+
— with zero third-party dependencies. No UI, no CLI, no MCP server.
|
|
14
|
+
|
|
15
|
+
> Need the CLI, UI, or MCP server? That's the
|
|
16
|
+
> [`odf`](https://opendataframework.github.io/odf/) package, which
|
|
17
|
+
> depends on this one.
|
|
18
|
+
|
|
19
|
+
Full documentation: **[opendataframework.github.io/opendataframework](https://opendataframework.github.io/opendataframework/)**
|
|
20
|
+
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
# Why
|
|
24
|
+
|
|
25
|
+
Most data projects solve the same structural problems repeatedly: how to
|
|
26
|
+
wire dependencies, manage component lifecycles, and keep configuration
|
|
27
|
+
separate from code. This framework provides that plumbing as a small set
|
|
28
|
+
of reusable abstractions — `Entity`, `Repository`, `Component`, `Service`,
|
|
29
|
+
`Task`, `Pipeline` — so it doesn't need to be reinvented by hand in every
|
|
30
|
+
project.
|
|
31
|
+
|
|
32
|
+
> Want a ready-made project layout instead? Scaffolding an actual project
|
|
33
|
+
> from a template is the [`odf`](https://opendataframework.github.io/odf/)
|
|
34
|
+
> package's job — this is what that scaffold is built on.
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
38
|
+
# Getting Started
|
|
39
|
+
|
|
40
|
+
Install:
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
pip install opendataframework
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
A minimal project needs three things: an entity and repository, a config
|
|
47
|
+
file, and a main file that starts the `Project`.
|
|
48
|
+
|
|
49
|
+
Application code (`app.py`):
|
|
50
|
+
|
|
51
|
+
```python
|
|
52
|
+
import sqlite3
|
|
53
|
+
from dataclasses import dataclass
|
|
54
|
+
|
|
55
|
+
from opendataframework import Component, Config, Entity, Repository
|
|
56
|
+
|
|
57
|
+
|
|
58
|
+
@Entity
|
|
59
|
+
@dataclass
|
|
60
|
+
class User:
|
|
61
|
+
id: int
|
|
62
|
+
name: str
|
|
63
|
+
|
|
64
|
+
|
|
65
|
+
@Component
|
|
66
|
+
class SQLite:
|
|
67
|
+
def __init__(self, config: Config):
|
|
68
|
+
self.conn = sqlite3.connect(config.sqlite.path)
|
|
69
|
+
self.conn.execute("CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT)")
|
|
70
|
+
|
|
71
|
+
|
|
72
|
+
@Repository(User)
|
|
73
|
+
class Users:
|
|
74
|
+
def __init__(self, sqlite: SQLite):
|
|
75
|
+
self.db = sqlite.conn
|
|
76
|
+
|
|
77
|
+
def all(self) -> list[User]:
|
|
78
|
+
rows = self.db.execute("SELECT id, name FROM users").fetchall()
|
|
79
|
+
return [User(id=row[0], name=row[1]) for row in rows]
|
|
80
|
+
|
|
81
|
+
def save(self, user: User) -> None:
|
|
82
|
+
self.db.execute("INSERT OR REPLACE INTO users VALUES (?, ?)", (user.id, user.name))
|
|
83
|
+
self.db.commit()
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Config file (`config.toml`):
|
|
87
|
+
|
|
88
|
+
```toml
|
|
89
|
+
[sqlite]
|
|
90
|
+
path = "app.db"
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Main file (`main.py`):
|
|
94
|
+
|
|
95
|
+
```python
|
|
96
|
+
from opendataframework import Project
|
|
97
|
+
|
|
98
|
+
from app import User, Users
|
|
99
|
+
|
|
100
|
+
project = Project.from_config("config.toml")
|
|
101
|
+
project.start()
|
|
102
|
+
|
|
103
|
+
users = project.context.get(Users)
|
|
104
|
+
users.save(User(id=1, name="Ada"))
|
|
105
|
+
|
|
106
|
+
print(users.all())
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
`project.start()` blocks until every component has completed its
|
|
110
|
+
initialisation stage — there is no hidden latency on first access. Adding
|
|
111
|
+
a new component means declaring its dependencies in the constructor, not
|
|
112
|
+
writing wiring code — a config section is only needed if the component
|
|
113
|
+
reads from `Config` itself.
|
|
114
|
+
|
|
115
|
+
---
|
|
116
|
+
|
|
117
|
+
# Core Concepts
|
|
118
|
+
|
|
119
|
+
## Entity
|
|
120
|
+
|
|
121
|
+
A structured unit of data the system works with.
|
|
122
|
+
|
|
123
|
+
```python
|
|
124
|
+
@Entity
|
|
125
|
+
@dataclass
|
|
126
|
+
class User:
|
|
127
|
+
id: int
|
|
128
|
+
name: str
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
## Repository
|
|
132
|
+
|
|
133
|
+
Manages Entities; abstracts persistence away from business logic.
|
|
134
|
+
|
|
135
|
+
```python
|
|
136
|
+
@Repository(User)
|
|
137
|
+
class Users:
|
|
138
|
+
def get(self, user_id: int) -> User: ...
|
|
139
|
+
def save(self, user: User) -> None: ...
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
## View
|
|
143
|
+
|
|
144
|
+
An optional, single declaration a `Repository` attaches to itself: which
|
|
145
|
+
representation (table, map, image, video, ...) and field(s) best fit its
|
|
146
|
+
data. Metadata only — this package renders nothing itself. See
|
|
147
|
+
[View](https://opendataframework.github.io/opendataframework/view/).
|
|
148
|
+
|
|
149
|
+
```python
|
|
150
|
+
@Repository(Store)
|
|
151
|
+
class Stores:
|
|
152
|
+
def all(self) -> list[Store]: ...
|
|
153
|
+
|
|
154
|
+
def data_view(self) -> LocationView:
|
|
155
|
+
return LocationView(fields=("lat", "lon"))
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
## Component
|
|
159
|
+
|
|
160
|
+
A `Context`-managed object with no required execution contract — wired in
|
|
161
|
+
and given its dependencies through the constructor.
|
|
162
|
+
|
|
163
|
+
```python
|
|
164
|
+
@Component
|
|
165
|
+
class Classifier:
|
|
166
|
+
def fit(self, data): ...
|
|
167
|
+
def predict(self, data): ...
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
## Service
|
|
171
|
+
|
|
172
|
+
A long-running executable, wired through DI the same way as a `Component` but
|
|
173
|
+
with a mandatory `setup → run → stop` lifecycle — an API server, worker
|
|
174
|
+
process, or scheduler. `run()` blocks by nature; the framework backgrounds it
|
|
175
|
+
automatically.
|
|
176
|
+
|
|
177
|
+
```python
|
|
178
|
+
@Service
|
|
179
|
+
class Postgres:
|
|
180
|
+
def setup(self): ...
|
|
181
|
+
def run(self): ...
|
|
182
|
+
def stop(self): ...
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
## Task
|
|
186
|
+
|
|
187
|
+
A finite executable, wired through DI the same way as a `Component` —
|
|
188
|
+
performs one bounded unit of work, once.
|
|
189
|
+
|
|
190
|
+
```python
|
|
191
|
+
@Task
|
|
192
|
+
class MetricsFetcher:
|
|
193
|
+
def execute(self): ...
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
## Pipeline
|
|
197
|
+
|
|
198
|
+
Coordinates multiple `Task`s (and optionally other `Pipeline`s) into an
|
|
199
|
+
ordered workflow.
|
|
200
|
+
|
|
201
|
+
```python
|
|
202
|
+
@Pipeline
|
|
203
|
+
class DailyAnalytics:
|
|
204
|
+
def execute(self): ...
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
## Layer
|
|
208
|
+
|
|
209
|
+
An organisational grouping — which subsystem a component belongs to
|
|
210
|
+
(`Api`, `Storage`, `Analytics`, `Messaging`, `Monitoring`, `Security`, or
|
|
211
|
+
custom). Independent of execution type.
|
|
212
|
+
|
|
213
|
+
```python
|
|
214
|
+
@Storage
|
|
215
|
+
@Service
|
|
216
|
+
class Postgres: ...
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
## Namespace
|
|
220
|
+
|
|
221
|
+
The base class underlying every framework decorator — `Entity`,
|
|
222
|
+
`Component`, `Service`, `Task`, `Pipeline`, `Repository`, and `Layer` are
|
|
223
|
+
all `Namespace` subclasses, each with its own independent name → class
|
|
224
|
+
mapping. A framework extension point, not something end users subclass
|
|
225
|
+
directly. See
|
|
226
|
+
[Namespace](https://opendataframework.github.io/opendataframework/namespace/).
|
|
227
|
+
|
|
228
|
+
## Config
|
|
229
|
+
|
|
230
|
+
Typed, dot-accessible view over a configuration dict — snake_case and
|
|
231
|
+
kebab-case are interchangeable, and nested sections chain naturally.
|
|
232
|
+
Resolved automatically into the container by `Project.from_config`.
|
|
233
|
+
|
|
234
|
+
```python
|
|
235
|
+
@Storage
|
|
236
|
+
@Service
|
|
237
|
+
class Postgres:
|
|
238
|
+
def __init__(self, config: Config):
|
|
239
|
+
self.host = config.postgres.host
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
## Logger
|
|
243
|
+
|
|
244
|
+
Per-component logging handle, injectable via constructor DI like any
|
|
245
|
+
other dependency — writes land in that component's own log file with no
|
|
246
|
+
name argument needed.
|
|
247
|
+
|
|
248
|
+
```python
|
|
249
|
+
@Task
|
|
250
|
+
class ExportUsers:
|
|
251
|
+
def __init__(self, users: Users, logger: Logger):
|
|
252
|
+
self.users = users
|
|
253
|
+
self.logger = logger
|
|
254
|
+
|
|
255
|
+
def execute(self) -> None:
|
|
256
|
+
self.logger.info("export starting")
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
## Context
|
|
260
|
+
|
|
261
|
+
Registers, resolves, and lifecycles every component in dependency order.
|
|
262
|
+
No circular dependencies — restructure with a shared third collaborator
|
|
263
|
+
instead.
|
|
264
|
+
|
|
265
|
+
```python
|
|
266
|
+
component = project.context.get(UsersApi) # typed, by class
|
|
267
|
+
```
|
|
268
|
+
|
|
269
|
+
## Project
|
|
270
|
+
|
|
271
|
+
The composition root — owns `Context` and configuration, and is the
|
|
272
|
+
single entry point for starting and stopping the application.
|
|
273
|
+
|
|
274
|
+
```python
|
|
275
|
+
project = Project.from_config("config.toml")
|
|
276
|
+
project.start()
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
Multiple independent `Project` instances are supported in the same
|
|
280
|
+
process; there is no global mutable state.
|
|
281
|
+
|
|
282
|
+
---
|
|
283
|
+
|
|
284
|
+
# Design Guidelines
|
|
285
|
+
|
|
286
|
+
* **Prefer composition** — decorators (`@Api @Service`), not inheritance
|
|
287
|
+
from framework base classes.
|
|
288
|
+
* **Keep responsibilities focused** — one primary reason to change per
|
|
289
|
+
class.
|
|
290
|
+
* **Avoid global mutable state** — framework state belongs to objects.
|
|
291
|
+
* **Avoid hidden magic** — dependencies must be visible in constructor
|
|
292
|
+
signatures.
|
|
293
|
+
|
|
294
|
+
---
|
|
295
|
+
|
|
296
|
+
# Mental Model
|
|
297
|
+
|
|
298
|
+
```text
|
|
299
|
+
Data is represented by Entities.
|
|
300
|
+
Repositories manage data.
|
|
301
|
+
|
|
302
|
+
Components provide capabilities.
|
|
303
|
+
Services run for extended periods.
|
|
304
|
+
Tasks execute bounded work.
|
|
305
|
+
Pipelines coordinate tasks (and other pipelines).
|
|
306
|
+
|
|
307
|
+
Layers organize components.
|
|
308
|
+
Context manages components and repositories.
|
|
309
|
+
Project owns everything.
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
---
|
|
313
|
+
|
|
314
|
+
# Examples
|
|
315
|
+
|
|
316
|
+
See [`examples/`](examples/README.md) — small, focused projects each
|
|
317
|
+
isolating one core abstraction.
|
|
318
|
+
|
|
319
|
+
> For the CLI, UI, MCP server, and chat surface
|
|
320
|
+
> built on top of this package, see `odf`.
|
|
321
|
+
|
|
322
|
+
---
|
|
323
|
+
|
|
324
|
+
# Development
|
|
325
|
+
|
|
326
|
+
Requires Python >=3.14, managed with [Poetry](https://python-poetry.org/).
|
|
327
|
+
|
|
328
|
+
```bash
|
|
329
|
+
poetry install # install dependencies, including dev
|
|
330
|
+
poetry run pytest # run the test suite
|
|
331
|
+
```
|
|
332
|
+
|
|
333
|
+
See [`CONTRIBUTING.md`](CONTRIBUTING.md) for how to propose changes.
|
|
334
|
+
|
|
335
|
+
---
|
|
336
|
+
|
|
337
|
+
# License
|
|
338
|
+
|
|
339
|
+
[MIT](LICENSE)
|