tinyorch 0.1.0__tar.gz → 0.1.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {tinyorch-0.1.0 → tinyorch-0.1.2}/PKG-INFO +56 -29
- tinyorch-0.1.2/README.md +144 -0
- {tinyorch-0.1.0 → tinyorch-0.1.2}/pyproject.toml +5 -3
- {tinyorch-0.1.0 → tinyorch-0.1.2}/pyproject.toml.orig +9 -3
- {tinyorch-0.1.0 → tinyorch-0.1.2}/src/tinyorch/_flow.py +2 -0
- {tinyorch-0.1.0 → tinyorch-0.1.2}/src/tinyorch/_task.py +2 -0
- tinyorch-0.1.2/src/tinyorch/_validate.py +32 -0
- tinyorch-0.1.0/README.md +0 -118
- {tinyorch-0.1.0 → tinyorch-0.1.2}/LICENSE +0 -0
- {tinyorch-0.1.0 → tinyorch-0.1.2}/src/tinyorch/__init__.py +0 -0
- {tinyorch-0.1.0 → tinyorch-0.1.2}/src/tinyorch/_api.py +0 -0
- {tinyorch-0.1.0 → tinyorch-0.1.2}/src/tinyorch/_context.py +0 -0
- {tinyorch-0.1.0 → tinyorch-0.1.2}/src/tinyorch/py.typed +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: tinyorch
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.2
|
|
4
4
|
Summary: A tiny, zero-dependency orchestrator for single-node, item-level pipelines.
|
|
5
5
|
Author: Murilegus
|
|
6
6
|
Author-email: Murilegus <murilegus@outlook.com>
|
|
@@ -10,6 +10,7 @@ Classifier: Development Status :: 3 - Alpha
|
|
|
10
10
|
Classifier: Programming Language :: Python :: 3
|
|
11
11
|
Classifier: Programming Language :: Python :: 3.12
|
|
12
12
|
Classifier: Programming Language :: Python :: 3.13
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
13
14
|
Classifier: Typing :: Typed
|
|
14
15
|
Requires-Python: >=3.12
|
|
15
16
|
Project-URL: Homepage, https://github.com/murilegus/tinyorch
|
|
@@ -21,14 +22,18 @@ Description-Content-Type: text/markdown
|
|
|
21
22
|
|
|
22
23
|
[](https://github.com/murilegus/tinyorch/actions/workflows/ci.yml)
|
|
23
24
|
[](https://pypi.org/project/tinyorch/)
|
|
24
|
-
[](LICENSE)
|
|
25
|
+
[](https://pypi.org/project/tinyorch/)
|
|
26
|
+
[](https://github.com/murilegus/tinyorch/blob/main/LICENSE)
|
|
26
27
|
|
|
27
28
|
A tiny, zero-dependency orchestrator for single-node, item-level pipelines.
|
|
28
29
|
|
|
29
|
-
🚧 **Status**:
|
|
30
|
-
tracking. No DAG, parallelism, retries or checkpoints yet; those are phase 2–3.
|
|
30
|
+
🚧 **Status**: alpha — phase 1 released (0.1), phase 2 in progress. See [Roadmap](#-roadmap).
|
|
31
31
|
|
|
32
|
+
> ⚠️ **What exists today (0.1):** `@task` / `@flow` decorators and in-memory run
|
|
33
|
+
> tracking — nothing more. Pipelining, bounded buffers, per-item resume and
|
|
34
|
+
> resources are **planned, not implemented**. The comparisons and performance
|
|
35
|
+
> claims below describe the design target; **none of them has been benchmarked
|
|
36
|
+
> yet**.
|
|
32
37
|
|
|
33
38
|
## 🚀 Quickstart
|
|
34
39
|
|
|
@@ -72,33 +77,42 @@ type checkers.
|
|
|
72
77
|
|
|
73
78
|
## 🤔 Why tinyorch?
|
|
74
79
|
|
|
75
|
-
tinyorch is
|
|
76
|
-
and too painful as a bare script**: a few thousand to a few million
|
|
77
|
-
(URLs, documents, records) flowing through I/O-heavy stages on one
|
|
78
|
-
where any stage can fail halfway.
|
|
80
|
+
tinyorch is aimed at the job that is **too small for Prefect, too finite for
|
|
81
|
+
Flink, and too painful as a bare script**: a few thousand to a few million
|
|
82
|
+
items (URLs, documents, records) flowing through I/O-heavy stages on one
|
|
83
|
+
machine, where any stage can fail halfway.
|
|
79
84
|
|
|
80
|
-
Three families of tools already exist for pipelines.
|
|
81
|
-
|
|
85
|
+
Three families of tools already exist for pipelines. The table below is how
|
|
86
|
+
tinyorch positions itself against them.
|
|
82
87
|
|
|
83
|
-
|
|
88
|
+
> ⚠️ **Unverified.** The other tools' columns are generalizations from their
|
|
89
|
+
> documentation, not from measurements or side-by-side use, and some cells are
|
|
90
|
+
> coarser than the tools deserve. The **tinyorch** column is the design target
|
|
91
|
+
> for phases 2–4; cells marked *(planned)* are not implemented in 0.1.
|
|
92
|
+
|
|
93
|
+
| | Workflow orchestrators<br>Airflow · Prefect · Dagster | Stream processors<br>Flink · Bytewax · Faust | Bare script<br>`asyncio.gather` + `tenacity` | **tinyorch** (target) |
|
|
84
94
|
|---|---|---|---|---|
|
|
85
95
|
| Input | bounded, per task | **unbounded**, per event | bounded | bounded, per item |
|
|
86
|
-
| Execution | one stage finishes before the next starts | continuous | ad hoc | pipelined, bounded buffers |
|
|
87
|
-
| Resume after crash | per task / per run | cluster checkpoints | hand-rolled or none | per item |
|
|
88
|
-
| Shared resources (browser, LLM client) | per process, awkward | n/a | manual | first-class |
|
|
96
|
+
| Execution | one stage finishes before the next starts | continuous | ad hoc | pipelined, bounded buffers *(planned, phase 2)* |
|
|
97
|
+
| Resume after crash | per task / per run | cluster checkpoints | hand-rolled or none | per item *(planned, phase 3)* |
|
|
98
|
+
| Shared resources (browser, LLM client) | per process, awkward | n/a | manual | first-class *(planned, phase 4)* |
|
|
89
99
|
| Infra | server + database | cluster | none | none |
|
|
90
|
-
| Sweet spot | multi-team, multi-machine DAGs | 24/7 event streams | one-off | one node, 10³–10⁶ items, crash-prone I/O |
|
|
100
|
+
| Sweet spot | multi-team, multi-machine DAGs | 24/7 event streams | one-off | one node, 10³–10⁶ items, crash-prone I/O *(unmeasured)* |
|
|
91
101
|
|
|
92
102
|
The architectural difference tinyorch is built around: in a stage-at-a-time
|
|
93
103
|
orchestrator, data crossing a task boundary is materialized (in RAM or an
|
|
94
|
-
object store) before the next stage starts. tinyorch
|
|
95
|
-
through stages with bounded buffers and
|
|
96
|
-
does what is left and memory stays flat regardless of
|
|
104
|
+
object store) before the next stage starts. tinyorch is designed to instead
|
|
105
|
+
pipeline items through stages with bounded buffers and checkpoint each item,
|
|
106
|
+
so that a rerun only does what is left and memory stays flat regardless of
|
|
107
|
+
input size. **This is a design goal, not a measured result**: none of it is
|
|
108
|
+
implemented in 0.1, and no benchmark backs the memory or resume claims yet.
|
|
109
|
+
|
|
110
|
+

|
|
97
111
|
|
|
98
|
-
|
|
112
|
+
*Conceptual diagram of the planned execution model, not a benchmark.*
|
|
99
113
|
|
|
100
|
-
**tinyorch is not a stream processor.** It
|
|
101
|
-
|
|
114
|
+
**tinyorch is not a stream processor.** It is meant to take a *bounded* input
|
|
115
|
+
and execute it with streaming semantics. There are no windows, no event time, no
|
|
102
116
|
unbounded sources, no distributed state. If you need those, use Flink.
|
|
103
117
|
|
|
104
118
|
**It is not a replacement for Prefect or Airflow either.** When the pipeline
|
|
@@ -108,11 +122,13 @@ aims to make that boundary obvious rather than blurry.
|
|
|
108
122
|
|
|
109
123
|
## 🏗️ Architecture
|
|
110
124
|
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
125
|
+
**Planned architecture** — today only the `@task` / `@flow` core exists; no
|
|
126
|
+
Protocols or adapters have been written yet. The intent: the core is
|
|
127
|
+
stdlib-only and defines Protocols; everything that touches the outside world
|
|
128
|
+
(state stores, browsers, LLM clients, CLI) is an adapter shipped as an
|
|
129
|
+
optional extra.
|
|
114
130
|
|
|
115
|
-

|
|
131
|
+

|
|
116
132
|
|
|
117
133
|
|
|
118
134
|
## 🚫 Non-goals
|
|
@@ -127,11 +143,22 @@ as an optional extra.
|
|
|
127
143
|
by schedule interval
|
|
128
144
|
|
|
129
145
|
|
|
130
|
-
##
|
|
146
|
+
## 🧭 Roadmap
|
|
131
147
|
|
|
132
|
-
|
|
148
|
+
| Phase | Scope | Status |
|
|
149
|
+
|---|---|---|
|
|
150
|
+
| 1 | `@task` / `@flow`, run tracking, typed public API | ✅ 0.1 |
|
|
151
|
+
| 2 | Pipelining: streaming tasks, bounded buffers, concurrent execution | 🚧 next |
|
|
152
|
+
| 3 | Per-item checkpoints, resumable runs, pluggable state store | planned |
|
|
153
|
+
| 4 | Resources with lifecycle and injection; process workers for CPU-bound tasks | planned |
|
|
154
|
+
| 5 | Dogfooding on a real crawl + LLM pipeline; CLI | planned |
|
|
133
155
|
|
|
156
|
+
Progress is tracked in [GitHub milestones](https://github.com/murilegus/tinyorch/milestones).
|
|
157
|
+
|
|
158
|
+
|
|
159
|
+
## 📄 Documentation
|
|
134
160
|
|
|
161
|
+
- [`docs/design/`](https://github.com/murilegus/tinyorch/tree/main/docs/design) — design decisions are recorded as ADRs.
|
|
135
162
|
## 🎫 License
|
|
136
163
|
|
|
137
|
-
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
|
|
164
|
+
This project is licensed under the MIT License - see the [LICENSE](https://github.com/murilegus/tinyorch/blob/main/LICENSE) file for details.
|
tinyorch-0.1.2/README.md
ADDED
|
@@ -0,0 +1,144 @@
|
|
|
1
|
+
# 🔀 tinyorch
|
|
2
|
+
|
|
3
|
+
[](https://github.com/murilegus/tinyorch/actions/workflows/ci.yml)
|
|
4
|
+
[](https://pypi.org/project/tinyorch/)
|
|
5
|
+
[](https://pypi.org/project/tinyorch/)
|
|
6
|
+
[](https://github.com/murilegus/tinyorch/blob/main/LICENSE)
|
|
7
|
+
|
|
8
|
+
A tiny, zero-dependency orchestrator for single-node, item-level pipelines.
|
|
9
|
+
|
|
10
|
+
🚧 **Status**: alpha — phase 1 released (0.1), phase 2 in progress. See [Roadmap](#-roadmap).
|
|
11
|
+
|
|
12
|
+
> ⚠️ **What exists today (0.1):** `@task` / `@flow` decorators and in-memory run
|
|
13
|
+
> tracking — nothing more. Pipelining, bounded buffers, per-item resume and
|
|
14
|
+
> resources are **planned, not implemented**. The comparisons and performance
|
|
15
|
+
> claims below describe the design target; **none of them has been benchmarked
|
|
16
|
+
> yet**.
|
|
17
|
+
|
|
18
|
+
## 🚀 Quickstart
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
pip install tinyorch
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
```python
|
|
25
|
+
import asyncio
|
|
26
|
+
from tinyorch import task, flow, current_run
|
|
27
|
+
|
|
28
|
+
|
|
29
|
+
@task
|
|
30
|
+
async def fetch(url: str) -> str:
|
|
31
|
+
await asyncio.sleep(0.1) # pretend to hit the network
|
|
32
|
+
return f"<html>{url}</html>"
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
@task
|
|
36
|
+
def parse(html: str) -> list[str]:
|
|
37
|
+
return html.split()
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
@flow
|
|
41
|
+
async def crawl(urls: list[str]) -> int:
|
|
42
|
+
for url in urls:
|
|
43
|
+
parse(await fetch(url))
|
|
44
|
+
run = current_run()
|
|
45
|
+
return len(run.calls) if run else 0
|
|
46
|
+
|
|
47
|
+
|
|
48
|
+
print(asyncio.run(crawl(["a", "b", "c"]))) # 6 task calls recorded
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
A `@task` is a plain callable with metadata: outside a flow it just runs.
|
|
52
|
+
Inside a `@flow`, every call is recorded on the current `RunContext`
|
|
53
|
+
(`current_run()`), which also tracks the flow's own timing and failure.
|
|
54
|
+
Sync and async functions both work; the wrapped signature is preserved for
|
|
55
|
+
type checkers.
|
|
56
|
+
|
|
57
|
+
|
|
58
|
+
## 🤔 Why tinyorch?
|
|
59
|
+
|
|
60
|
+
tinyorch is aimed at the job that is **too small for Prefect, too finite for
|
|
61
|
+
Flink, and too painful as a bare script**: a few thousand to a few million
|
|
62
|
+
items (URLs, documents, records) flowing through I/O-heavy stages on one
|
|
63
|
+
machine, where any stage can fail halfway.
|
|
64
|
+
|
|
65
|
+
Three families of tools already exist for pipelines. The table below is how
|
|
66
|
+
tinyorch positions itself against them.
|
|
67
|
+
|
|
68
|
+
> ⚠️ **Unverified.** The other tools' columns are generalizations from their
|
|
69
|
+
> documentation, not from measurements or side-by-side use, and some cells are
|
|
70
|
+
> coarser than the tools deserve. The **tinyorch** column is the design target
|
|
71
|
+
> for phases 2–4; cells marked *(planned)* are not implemented in 0.1.
|
|
72
|
+
|
|
73
|
+
| | Workflow orchestrators<br>Airflow · Prefect · Dagster | Stream processors<br>Flink · Bytewax · Faust | Bare script<br>`asyncio.gather` + `tenacity` | **tinyorch** (target) |
|
|
74
|
+
|---|---|---|---|---|
|
|
75
|
+
| Input | bounded, per task | **unbounded**, per event | bounded | bounded, per item |
|
|
76
|
+
| Execution | one stage finishes before the next starts | continuous | ad hoc | pipelined, bounded buffers *(planned, phase 2)* |
|
|
77
|
+
| Resume after crash | per task / per run | cluster checkpoints | hand-rolled or none | per item *(planned, phase 3)* |
|
|
78
|
+
| Shared resources (browser, LLM client) | per process, awkward | n/a | manual | first-class *(planned, phase 4)* |
|
|
79
|
+
| Infra | server + database | cluster | none | none |
|
|
80
|
+
| Sweet spot | multi-team, multi-machine DAGs | 24/7 event streams | one-off | one node, 10³–10⁶ items, crash-prone I/O *(unmeasured)* |
|
|
81
|
+
|
|
82
|
+
The architectural difference tinyorch is built around: in a stage-at-a-time
|
|
83
|
+
orchestrator, data crossing a task boundary is materialized (in RAM or an
|
|
84
|
+
object store) before the next stage starts. tinyorch is designed to instead
|
|
85
|
+
pipeline items through stages with bounded buffers and checkpoint each item,
|
|
86
|
+
so that a rerun only does what is left and memory stays flat regardless of
|
|
87
|
+
input size. **This is a design goal, not a measured result**: none of it is
|
|
88
|
+
implemented in 0.1, and no benchmark backs the memory or resume claims yet.
|
|
89
|
+
|
|
90
|
+

|
|
91
|
+
|
|
92
|
+
*Conceptual diagram of the planned execution model, not a benchmark.*
|
|
93
|
+
|
|
94
|
+
**tinyorch is not a stream processor.** It is meant to take a *bounded* input
|
|
95
|
+
and execute it with streaming semantics. There are no windows, no event time, no
|
|
96
|
+
unbounded sources, no distributed state. If you need those, use Flink.
|
|
97
|
+
|
|
98
|
+
**It is not a replacement for Prefect or Airflow either.** When the pipeline
|
|
99
|
+
outgrows one machine or one team, those are the tools to move to; tinyorch
|
|
100
|
+
aims to make that boundary obvious rather than blurry.
|
|
101
|
+
|
|
102
|
+
|
|
103
|
+
## 🏗️ Architecture
|
|
104
|
+
|
|
105
|
+
**Planned architecture** — today only the `@task` / `@flow` core exists; no
|
|
106
|
+
Protocols or adapters have been written yet. The intent: the core is
|
|
107
|
+
stdlib-only and defines Protocols; everything that touches the outside world
|
|
108
|
+
(state stores, browsers, LLM clients, CLI) is an adapter shipped as an
|
|
109
|
+
optional extra.
|
|
110
|
+
|
|
111
|
+

|
|
112
|
+
|
|
113
|
+
|
|
114
|
+
## 🚫 Non-goals
|
|
115
|
+
|
|
116
|
+
- **Multi-node scheduling** or a long-running scheduler service — one process,
|
|
117
|
+
one machine, started by cron, a Cloud Run Job or your own code
|
|
118
|
+
- **Unbounded / event-time stream processing** — input is finite; no windows,
|
|
119
|
+
watermarks or distributed state
|
|
120
|
+
- **A full web UI** with authentication and multi-tenancy — a local, read-only
|
|
121
|
+
run viewer is planned as an optional extra
|
|
122
|
+
- **Time-based backfills** in the Airflow sense — runs are keyed by input, not
|
|
123
|
+
by schedule interval
|
|
124
|
+
|
|
125
|
+
|
|
126
|
+
## 🧭 Roadmap
|
|
127
|
+
|
|
128
|
+
| Phase | Scope | Status |
|
|
129
|
+
|---|---|---|
|
|
130
|
+
| 1 | `@task` / `@flow`, run tracking, typed public API | ✅ 0.1 |
|
|
131
|
+
| 2 | Pipelining: streaming tasks, bounded buffers, concurrent execution | 🚧 next |
|
|
132
|
+
| 3 | Per-item checkpoints, resumable runs, pluggable state store | planned |
|
|
133
|
+
| 4 | Resources with lifecycle and injection; process workers for CPU-bound tasks | planned |
|
|
134
|
+
| 5 | Dogfooding on a real crawl + LLM pipeline; CLI | planned |
|
|
135
|
+
|
|
136
|
+
Progress is tracked in [GitHub milestones](https://github.com/murilegus/tinyorch/milestones).
|
|
137
|
+
|
|
138
|
+
|
|
139
|
+
## 📄 Documentation
|
|
140
|
+
|
|
141
|
+
- [`docs/design/`](https://github.com/murilegus/tinyorch/tree/main/docs/design) — design decisions are recorded as ADRs.
|
|
142
|
+
## 🎫 License
|
|
143
|
+
|
|
144
|
+
This project is licensed under the MIT License - see the [LICENSE](https://github.com/murilegus/tinyorch/blob/main/LICENSE) file for details.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
[project]
|
|
2
2
|
name = "tinyorch"
|
|
3
|
-
version = "0.1.
|
|
3
|
+
version = "0.1.2"
|
|
4
4
|
description = "A tiny, zero-dependency orchestrator for single-node, item-level pipelines."
|
|
5
5
|
readme = "README.md"
|
|
6
6
|
license = "MIT"
|
|
@@ -11,6 +11,7 @@ classifiers = [
|
|
|
11
11
|
"Programming Language :: Python :: 3",
|
|
12
12
|
"Programming Language :: Python :: 3.12",
|
|
13
13
|
"Programming Language :: Python :: 3.13",
|
|
14
|
+
"Programming Language :: Python :: 3.14",
|
|
14
15
|
"Typing :: Typed",
|
|
15
16
|
]
|
|
16
17
|
dependencies = []
|
|
@@ -44,6 +45,7 @@ include = [
|
|
|
44
45
|
]
|
|
45
46
|
typeCheckingMode = "strict"
|
|
46
47
|
failOnWarnings = true
|
|
48
|
+
reportUnnecessaryTypeIgnoreComment = "error"
|
|
47
49
|
|
|
48
50
|
[[tool.basedpyright.executionEnvironments]]
|
|
49
51
|
root = "tests"
|
|
@@ -51,7 +53,7 @@ root = "tests"
|
|
|
51
53
|
[tool.pytest.ini_options]
|
|
52
54
|
asyncio_mode = "auto"
|
|
53
55
|
testpaths = ["tests"]
|
|
54
|
-
addopts = "--cov=tinyorch --cov-report=term-missing
|
|
56
|
+
addopts = "--cov=tinyorch --cov-report=term-missing"
|
|
55
57
|
|
|
56
58
|
[tool.coverage.run]
|
|
57
59
|
branch = true
|
|
@@ -61,7 +63,7 @@ source = ["tinyorch"]
|
|
|
61
63
|
exclude_also = [
|
|
62
64
|
"if TYPE_CHECKING:",
|
|
63
65
|
"@overload",
|
|
64
|
-
'
|
|
66
|
+
'^\s*\.\.\.$',
|
|
65
67
|
]
|
|
66
68
|
|
|
67
69
|
[tool.ruff]
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
[project]
|
|
2
2
|
name = "tinyorch"
|
|
3
|
-
version = "0.1.
|
|
3
|
+
version = "0.1.2"
|
|
4
4
|
description = "A tiny, zero-dependency orchestrator for single-node, item-level pipelines."
|
|
5
5
|
readme = "README.md"
|
|
6
6
|
authors = [{ name = "Murilegus", email = "murilegus@outlook.com" }]
|
|
@@ -12,6 +12,7 @@ classifiers = [
|
|
|
12
12
|
"Programming Language :: Python :: 3",
|
|
13
13
|
"Programming Language :: Python :: 3.12",
|
|
14
14
|
"Programming Language :: Python :: 3.13",
|
|
15
|
+
"Programming Language :: Python :: 3.14",
|
|
15
16
|
"Typing :: Typed",
|
|
16
17
|
]
|
|
17
18
|
dependencies = []
|
|
@@ -38,6 +39,9 @@ build-backend = "uv_build"
|
|
|
38
39
|
include = ["src", "tests"]
|
|
39
40
|
typeCheckingMode = "strict"
|
|
40
41
|
failOnWarnings = true
|
|
42
|
+
# Off by default even in strict; without it a stale `# pyright: ignore[...]`
|
|
43
|
+
# passes silently and the negative checks in tests/test_typing.py guard nothing.
|
|
44
|
+
reportUnnecessaryTypeIgnoreComment = "error"
|
|
41
45
|
|
|
42
46
|
[[tool.basedpyright.executionEnvironments]]
|
|
43
47
|
root = "tests"
|
|
@@ -45,7 +49,9 @@ root = "tests"
|
|
|
45
49
|
[tool.pytest.ini_options]
|
|
46
50
|
asyncio_mode = "auto"
|
|
47
51
|
testpaths = ["tests"]
|
|
48
|
-
|
|
52
|
+
# The coverage floor is enforced in CI (`pytest --cov-fail-under=95`), not here,
|
|
53
|
+
# so running a single test file locally does not fail on partial coverage.
|
|
54
|
+
addopts = "--cov=tinyorch --cov-report=term-missing"
|
|
49
55
|
|
|
50
56
|
[tool.coverage.run]
|
|
51
57
|
branch = true
|
|
@@ -55,7 +61,7 @@ source = ["tinyorch"]
|
|
|
55
61
|
exclude_also = [
|
|
56
62
|
"if TYPE_CHECKING:",
|
|
57
63
|
"@overload",
|
|
58
|
-
"
|
|
64
|
+
"^\\s*\\.\\.\\.$",
|
|
59
65
|
]
|
|
60
66
|
|
|
61
67
|
[tool.ruff]
|
|
@@ -7,6 +7,7 @@ import inspect
|
|
|
7
7
|
from typing import TYPE_CHECKING, cast, override
|
|
8
8
|
|
|
9
9
|
from tinyorch._context import enter_run
|
|
10
|
+
from tinyorch._validate import check_wrappable
|
|
10
11
|
|
|
11
12
|
if TYPE_CHECKING:
|
|
12
13
|
from collections.abc import Awaitable, Callable
|
|
@@ -14,6 +15,7 @@ if TYPE_CHECKING:
|
|
|
14
15
|
|
|
15
16
|
class Flow[**P, R]:
|
|
16
17
|
def __init__(self, fn: Callable[P, R], *, name: str | None = None) -> None:
|
|
18
|
+
check_wrappable(fn, "flow")
|
|
17
19
|
self.fn: Callable[P, R] = fn
|
|
18
20
|
self.name: str = name or fn.__name__
|
|
19
21
|
self.is_async: bool = inspect.iscoroutinefunction(fn)
|
|
@@ -8,6 +8,7 @@ import time
|
|
|
8
8
|
from typing import TYPE_CHECKING, cast, override
|
|
9
9
|
|
|
10
10
|
from tinyorch._context import RunContext, TaskCall, current_run
|
|
11
|
+
from tinyorch._validate import check_wrappable
|
|
11
12
|
|
|
12
13
|
if TYPE_CHECKING:
|
|
13
14
|
from collections.abc import Awaitable, Callable
|
|
@@ -15,6 +16,7 @@ if TYPE_CHECKING:
|
|
|
15
16
|
|
|
16
17
|
class Task[**P, R]:
|
|
17
18
|
def __init__(self, fn: Callable[P, R], *, name: str | None = None) -> None:
|
|
19
|
+
check_wrappable(fn, "task")
|
|
18
20
|
self.fn: Callable[P, R] = fn
|
|
19
21
|
self.name: str = name or fn.__name__
|
|
20
22
|
self.is_async: bool = inspect.iscoroutinefunction(fn)
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# SPDX-FileCopyrightText: © 2026 Murilegus <murilegus@outlook.com>
|
|
2
|
+
# SPDX-License-Identifier: MIT
|
|
3
|
+
|
|
4
|
+
from __future__ import annotations
|
|
5
|
+
|
|
6
|
+
import inspect
|
|
7
|
+
from typing import TYPE_CHECKING
|
|
8
|
+
|
|
9
|
+
if TYPE_CHECKING:
|
|
10
|
+
from collections.abc import Callable
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
def check_wrappable(fn: Callable[..., object], kind: str) -> None:
|
|
14
|
+
"""Reject callables whose body would not run inside the bookkeeping.
|
|
15
|
+
|
|
16
|
+
Only plain ``def`` / ``async def`` functions and bound methods are
|
|
17
|
+
accepted. See ADR-002.
|
|
18
|
+
"""
|
|
19
|
+
if not (inspect.isfunction(fn) or inspect.ismethod(fn)):
|
|
20
|
+
msg = (
|
|
21
|
+
f"@{kind} expects a function defined with 'def' or 'async def', "
|
|
22
|
+
f"got {type(fn).__name__}; wrap it in a function first"
|
|
23
|
+
)
|
|
24
|
+
raise TypeError(msg)
|
|
25
|
+
|
|
26
|
+
if inspect.isgeneratorfunction(fn) or inspect.isasyncgenfunction(fn):
|
|
27
|
+
msg = (
|
|
28
|
+
f"@{kind} does not support generator functions ({fn.__qualname__}): "
|
|
29
|
+
f"the body would run after the {kind} call has already returned. "
|
|
30
|
+
"Return a value from a plain 'def' or 'async def' instead"
|
|
31
|
+
)
|
|
32
|
+
raise TypeError(msg)
|
tinyorch-0.1.0/README.md
DELETED
|
@@ -1,118 +0,0 @@
|
|
|
1
|
-
# 🔀 tinyorch
|
|
2
|
-
|
|
3
|
-
[](https://github.com/murilegus/tinyorch/actions/workflows/ci.yml)
|
|
4
|
-
[](https://pypi.org/project/tinyorch/)
|
|
5
|
-
[](pyproject.toml)
|
|
6
|
-
[](LICENSE)
|
|
7
|
-
|
|
8
|
-
A tiny, zero-dependency orchestrator for single-node, item-level pipelines.
|
|
9
|
-
|
|
10
|
-
🚧 **Status**: Phase 1 complete — `@task` / `@flow` run sequentially with run
|
|
11
|
-
tracking. No DAG, parallelism, retries or checkpoints yet; those are phase 2–3.
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
## 🚀 Quickstart
|
|
15
|
-
|
|
16
|
-
```bash
|
|
17
|
-
pip install tinyorch
|
|
18
|
-
```
|
|
19
|
-
|
|
20
|
-
```python
|
|
21
|
-
import asyncio
|
|
22
|
-
from tinyorch import task, flow, current_run
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
@task
|
|
26
|
-
async def fetch(url: str) -> str:
|
|
27
|
-
await asyncio.sleep(0.1) # pretend to hit the network
|
|
28
|
-
return f"<html>{url}</html>"
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
@task
|
|
32
|
-
def parse(html: str) -> list[str]:
|
|
33
|
-
return html.split()
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
@flow
|
|
37
|
-
async def crawl(urls: list[str]) -> int:
|
|
38
|
-
for url in urls:
|
|
39
|
-
parse(await fetch(url))
|
|
40
|
-
run = current_run()
|
|
41
|
-
return len(run.calls) if run else 0
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
print(asyncio.run(crawl(["a", "b", "c"]))) # 6 task calls recorded
|
|
45
|
-
```
|
|
46
|
-
|
|
47
|
-
A `@task` is a plain callable with metadata: outside a flow it just runs.
|
|
48
|
-
Inside a `@flow`, every call is recorded on the current `RunContext`
|
|
49
|
-
(`current_run()`), which also tracks the flow's own timing and failure.
|
|
50
|
-
Sync and async functions both work; the wrapped signature is preserved for
|
|
51
|
-
type checkers.
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
## 🤔 Why tinyorch?
|
|
55
|
-
|
|
56
|
-
tinyorch is for the job that is **too small for Prefect, too finite for Flink,
|
|
57
|
-
and too painful as a bare script**: a few thousand to a few million items
|
|
58
|
-
(URLs, documents, records) flowing through I/O-heavy stages on one machine,
|
|
59
|
-
where any stage can fail halfway.
|
|
60
|
-
|
|
61
|
-
Three families of tools already exist for pipelines. Each is right for its
|
|
62
|
-
sweet spot and awkward outside it:
|
|
63
|
-
|
|
64
|
-
| | Workflow orchestrators<br>Airflow · Prefect · Dagster | Stream processors<br>Flink · Bytewax · Faust | Bare script<br>`asyncio.gather` + `tenacity` | **tinyorch** |
|
|
65
|
-
|---|---|---|---|---|
|
|
66
|
-
| Input | bounded, per task | **unbounded**, per event | bounded | bounded, per item |
|
|
67
|
-
| Execution | one stage finishes before the next starts | continuous | ad hoc | pipelined, bounded buffers |
|
|
68
|
-
| Resume after crash | per task / per run | cluster checkpoints | hand-rolled or none | per item |
|
|
69
|
-
| Shared resources (browser, LLM client) | per process, awkward | n/a | manual | first-class |
|
|
70
|
-
| Infra | server + database | cluster | none | none |
|
|
71
|
-
| Sweet spot | multi-team, multi-machine DAGs | 24/7 event streams | one-off | one node, 10³–10⁶ items, crash-prone I/O |
|
|
72
|
-
|
|
73
|
-
The architectural difference tinyorch is built around: in a stage-at-a-time
|
|
74
|
-
orchestrator, data crossing a task boundary is materialized (in RAM or an
|
|
75
|
-
object store) before the next stage starts. tinyorch instead pipelines items
|
|
76
|
-
through stages with bounded buffers and checkpoints each item, so a rerun only
|
|
77
|
-
does what is left and memory stays flat regardless of input size.
|
|
78
|
-
|
|
79
|
-

|
|
80
|
-
|
|
81
|
-
**tinyorch is not a stream processor.** It takes a *bounded* input and
|
|
82
|
-
executes it with streaming semantics. There are no windows, no event time, no
|
|
83
|
-
unbounded sources, no distributed state. If you need those, use Flink.
|
|
84
|
-
|
|
85
|
-
**It is not a replacement for Prefect or Airflow either.** When the pipeline
|
|
86
|
-
outgrows one machine or one team, those are the tools to move to; tinyorch
|
|
87
|
-
aims to make that boundary obvious rather than blurry.
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
## 🏗️ Architecture
|
|
91
|
-
|
|
92
|
-
The core is stdlib-only and defines Protocols; everything that touches the
|
|
93
|
-
outside world (state stores, browsers, LLM clients, CLI) is an adapter shipped
|
|
94
|
-
as an optional extra.
|
|
95
|
-
|
|
96
|
-

|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
## 🚫 Non-goals
|
|
100
|
-
|
|
101
|
-
- **Multi-node scheduling** or a long-running scheduler service — one process,
|
|
102
|
-
one machine, started by cron, a Cloud Run Job or your own code
|
|
103
|
-
- **Unbounded / event-time stream processing** — input is finite; no windows,
|
|
104
|
-
watermarks or distributed state
|
|
105
|
-
- **A full web UI** with authentication and multi-tenancy — a local, read-only
|
|
106
|
-
run viewer is planned as an optional extra
|
|
107
|
-
- **Time-based backfills** in the Airflow sense — runs are keyed by input, not
|
|
108
|
-
by schedule interval
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
## 📄 Documentation
|
|
112
|
-
|
|
113
|
-
- [`docs/design/`](docs/design/) — design decisions are recorded as ADRs.
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
## 🎫 License
|
|
117
|
-
|
|
118
|
-
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|