tinyorch 0.1.0__tar.gz → 0.1.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: tinyorch
3
- Version: 0.1.0
3
+ Version: 0.1.2
4
4
  Summary: A tiny, zero-dependency orchestrator for single-node, item-level pipelines.
5
5
  Author: Murilegus
6
6
  Author-email: Murilegus <murilegus@outlook.com>
@@ -10,6 +10,7 @@ Classifier: Development Status :: 3 - Alpha
10
10
  Classifier: Programming Language :: Python :: 3
11
11
  Classifier: Programming Language :: Python :: 3.12
12
12
  Classifier: Programming Language :: Python :: 3.13
13
+ Classifier: Programming Language :: Python :: 3.14
13
14
  Classifier: Typing :: Typed
14
15
  Requires-Python: >=3.12
15
16
  Project-URL: Homepage, https://github.com/murilegus/tinyorch
@@ -21,14 +22,18 @@ Description-Content-Type: text/markdown
21
22
 
22
23
  [![ci](https://github.com/murilegus/tinyorch/actions/workflows/ci.yml/badge.svg)](https://github.com/murilegus/tinyorch/actions/workflows/ci.yml)
23
24
  [![PyPI](https://img.shields.io/pypi/v/tinyorch)](https://pypi.org/project/tinyorch/)
24
- [![Python](https://img.shields.io/badge/python-3.12%20%7C%203.13-blue)](pyproject.toml)
25
- [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
25
+ [![Python](https://img.shields.io/pypi/pyversions/tinyorch)](https://pypi.org/project/tinyorch/)
26
+ [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/murilegus/tinyorch/blob/main/LICENSE)
26
27
 
27
28
  A tiny, zero-dependency orchestrator for single-node, item-level pipelines.
28
29
 
29
- 🚧 **Status**: Phase 1 complete — `@task` / `@flow` run sequentially with run
30
- tracking. No DAG, parallelism, retries or checkpoints yet; those are phase 2–3.
30
+ 🚧 **Status**: alpha — phase 1 released (0.1), phase 2 in progress. See [Roadmap](#-roadmap).
31
31
 
32
+ > ⚠️ **What exists today (0.1):** `@task` / `@flow` decorators and in-memory run
33
+ > tracking — nothing more. Pipelining, bounded buffers, per-item resume and
34
+ > resources are **planned, not implemented**. The comparisons and performance
35
+ > claims below describe the design target; **none of them has been benchmarked
36
+ > yet**.
32
37
 
33
38
  ## 🚀 Quickstart
34
39
 
@@ -72,33 +77,42 @@ type checkers.
72
77
 
73
78
  ## 🤔 Why tinyorch?
74
79
 
75
- tinyorch is for the job that is **too small for Prefect, too finite for Flink,
76
- and too painful as a bare script**: a few thousand to a few million items
77
- (URLs, documents, records) flowing through I/O-heavy stages on one machine,
78
- where any stage can fail halfway.
80
+ tinyorch is aimed at the job that is **too small for Prefect, too finite for
81
+ Flink, and too painful as a bare script**: a few thousand to a few million
82
+ items (URLs, documents, records) flowing through I/O-heavy stages on one
83
+ machine, where any stage can fail halfway.
79
84
 
80
- Three families of tools already exist for pipelines. Each is right for its
81
- sweet spot and awkward outside it:
85
+ Three families of tools already exist for pipelines. The table below is how
86
+ tinyorch positions itself against them.
82
87
 
83
- | | Workflow orchestrators<br>Airflow · Prefect · Dagster | Stream processors<br>Flink · Bytewax · Faust | Bare script<br>`asyncio.gather` + `tenacity` | **tinyorch** |
88
+ > ⚠️ **Unverified.** The other tools' columns are generalizations from their
89
+ > documentation, not from measurements or side-by-side use, and some cells are
90
+ > coarser than the tools deserve. The **tinyorch** column is the design target
91
+ > for phases 2–4; cells marked *(planned)* are not implemented in 0.1.
92
+
93
+ | | Workflow orchestrators<br>Airflow · Prefect · Dagster | Stream processors<br>Flink · Bytewax · Faust | Bare script<br>`asyncio.gather` + `tenacity` | **tinyorch** (target) |
84
94
  |---|---|---|---|---|
85
95
  | Input | bounded, per task | **unbounded**, per event | bounded | bounded, per item |
86
- | Execution | one stage finishes before the next starts | continuous | ad hoc | pipelined, bounded buffers |
87
- | Resume after crash | per task / per run | cluster checkpoints | hand-rolled or none | per item |
88
- | Shared resources (browser, LLM client) | per process, awkward | n/a | manual | first-class |
96
+ | Execution | one stage finishes before the next starts | continuous | ad hoc | pipelined, bounded buffers *(planned, phase 2)* |
97
+ | Resume after crash | per task / per run | cluster checkpoints | hand-rolled or none | per item *(planned, phase 3)* |
98
+ | Shared resources (browser, LLM client) | per process, awkward | n/a | manual | first-class *(planned, phase 4)* |
89
99
  | Infra | server + database | cluster | none | none |
90
- | Sweet spot | multi-team, multi-machine DAGs | 24/7 event streams | one-off | one node, 10³–10⁶ items, crash-prone I/O |
100
+ | Sweet spot | multi-team, multi-machine DAGs | 24/7 event streams | one-off | one node, 10³–10⁶ items, crash-prone I/O *(unmeasured)* |
91
101
 
92
102
  The architectural difference tinyorch is built around: in a stage-at-a-time
93
103
  orchestrator, data crossing a task boundary is materialized (in RAM or an
94
- object store) before the next stage starts. tinyorch instead pipelines items
95
- through stages with bounded buffers and checkpoints each item, so a rerun only
96
- does what is left and memory stays flat regardless of input size.
104
+ object store) before the next stage starts. tinyorch is designed to instead
105
+ pipeline items through stages with bounded buffers and checkpoint each item,
106
+ so that a rerun only does what is left and memory stays flat regardless of
107
+ input size. **This is a design goal, not a measured result**: none of it is
108
+ implemented in 0.1, and no benchmark backs the memory or resume claims yet.
109
+
110
+ ![Stage-at-a-time orchestrators vs tinyorch pipelining](https://raw.githubusercontent.com/murilegus/tinyorch/main/docs/assets/stage-vs-pipelined.svg)
97
111
 
98
- ![Stage-at-a-time orchestrators vs tinyorch pipelining](docs/assets/stage-vs-pipelined.svg)
112
+ *Conceptual diagram of the planned execution model, not a benchmark.*
99
113
 
100
- **tinyorch is not a stream processor.** It takes a *bounded* input and
101
- executes it with streaming semantics. There are no windows, no event time, no
114
+ **tinyorch is not a stream processor.** It is meant to take a *bounded* input
115
+ and execute it with streaming semantics. There are no windows, no event time, no
102
116
  unbounded sources, no distributed state. If you need those, use Flink.
103
117
 
104
118
  **It is not a replacement for Prefect or Airflow either.** When the pipeline
@@ -108,11 +122,13 @@ aims to make that boundary obvious rather than blurry.
108
122
 
109
123
  ## 🏗️ Architecture
110
124
 
111
- The core is stdlib-only and defines Protocols; everything that touches the
112
- outside world (state stores, browsers, LLM clients, CLI) is an adapter shipped
113
- as an optional extra.
125
+ **Planned architecture** — today only the `@task` / `@flow` core exists; no
126
+ Protocols or adapters have been written yet. The intent: the core is
127
+ stdlib-only and defines Protocols; everything that touches the outside world
128
+ (state stores, browsers, LLM clients, CLI) is an adapter shipped as an
129
+ optional extra.
114
130
 
115
- ![tinyorch architecture](docs/assets/architecture.svg)
131
+ ![tinyorch architecture](https://raw.githubusercontent.com/murilegus/tinyorch/main/docs/assets/architecture.svg)
116
132
 
117
133
 
118
134
  ## 🚫 Non-goals
@@ -127,11 +143,22 @@ as an optional extra.
127
143
  by schedule interval
128
144
 
129
145
 
130
- ## 📄 Documentation
146
+ ## 🧭 Roadmap
131
147
 
132
- - [`docs/design/`](docs/design/) — design decisions are recorded as ADRs.
148
+ | Phase | Scope | Status |
149
+ |---|---|---|
150
+ | 1 | `@task` / `@flow`, run tracking, typed public API | ✅ 0.1 |
151
+ | 2 | Pipelining: streaming tasks, bounded buffers, concurrent execution | 🚧 next |
152
+ | 3 | Per-item checkpoints, resumable runs, pluggable state store | planned |
153
+ | 4 | Resources with lifecycle and injection; process workers for CPU-bound tasks | planned |
154
+ | 5 | Dogfooding on a real crawl + LLM pipeline; CLI | planned |
133
155
 
156
+ Progress is tracked in [GitHub milestones](https://github.com/murilegus/tinyorch/milestones).
157
+
158
+
159
+ ## 📄 Documentation
134
160
 
161
+ - [`docs/design/`](https://github.com/murilegus/tinyorch/tree/main/docs/design) — design decisions are recorded as ADRs.
135
162
  ## 🎫 License
136
163
 
137
- This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
164
+ This project is licensed under the MIT License - see the [LICENSE](https://github.com/murilegus/tinyorch/blob/main/LICENSE) file for details.
@@ -0,0 +1,144 @@
1
+ # 🔀 tinyorch
2
+
3
+ [![ci](https://github.com/murilegus/tinyorch/actions/workflows/ci.yml/badge.svg)](https://github.com/murilegus/tinyorch/actions/workflows/ci.yml)
4
+ [![PyPI](https://img.shields.io/pypi/v/tinyorch)](https://pypi.org/project/tinyorch/)
5
+ [![Python](https://img.shields.io/pypi/pyversions/tinyorch)](https://pypi.org/project/tinyorch/)
6
+ [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/murilegus/tinyorch/blob/main/LICENSE)
7
+
8
+ A tiny, zero-dependency orchestrator for single-node, item-level pipelines.
9
+
10
+ 🚧 **Status**: alpha — phase 1 released (0.1), phase 2 in progress. See [Roadmap](#-roadmap).
11
+
12
+ > ⚠️ **What exists today (0.1):** `@task` / `@flow` decorators and in-memory run
13
+ > tracking — nothing more. Pipelining, bounded buffers, per-item resume and
14
+ > resources are **planned, not implemented**. The comparisons and performance
15
+ > claims below describe the design target; **none of them has been benchmarked
16
+ > yet**.
17
+
18
+ ## 🚀 Quickstart
19
+
20
+ ```bash
21
+ pip install tinyorch
22
+ ```
23
+
24
+ ```python
25
+ import asyncio
26
+ from tinyorch import task, flow, current_run
27
+
28
+
29
+ @task
30
+ async def fetch(url: str) -> str:
31
+ await asyncio.sleep(0.1) # pretend to hit the network
32
+ return f"<html>{url}</html>"
33
+
34
+
35
+ @task
36
+ def parse(html: str) -> list[str]:
37
+ return html.split()
38
+
39
+
40
+ @flow
41
+ async def crawl(urls: list[str]) -> int:
42
+ for url in urls:
43
+ parse(await fetch(url))
44
+ run = current_run()
45
+ return len(run.calls) if run else 0
46
+
47
+
48
+ print(asyncio.run(crawl(["a", "b", "c"]))) # 6 task calls recorded
49
+ ```
50
+
51
+ A `@task` is a plain callable with metadata: outside a flow it just runs.
52
+ Inside a `@flow`, every call is recorded on the current `RunContext`
53
+ (`current_run()`), which also tracks the flow's own timing and failure.
54
+ Sync and async functions both work; the wrapped signature is preserved for
55
+ type checkers.
56
+
57
+
58
+ ## 🤔 Why tinyorch?
59
+
60
+ tinyorch is aimed at the job that is **too small for Prefect, too finite for
61
+ Flink, and too painful as a bare script**: a few thousand to a few million
62
+ items (URLs, documents, records) flowing through I/O-heavy stages on one
63
+ machine, where any stage can fail halfway.
64
+
65
+ Three families of tools already exist for pipelines. The table below is how
66
+ tinyorch positions itself against them.
67
+
68
+ > ⚠️ **Unverified.** The other tools' columns are generalizations from their
69
+ > documentation, not from measurements or side-by-side use, and some cells are
70
+ > coarser than the tools deserve. The **tinyorch** column is the design target
71
+ > for phases 2–4; cells marked *(planned)* are not implemented in 0.1.
72
+
73
+ | | Workflow orchestrators<br>Airflow · Prefect · Dagster | Stream processors<br>Flink · Bytewax · Faust | Bare script<br>`asyncio.gather` + `tenacity` | **tinyorch** (target) |
74
+ |---|---|---|---|---|
75
+ | Input | bounded, per task | **unbounded**, per event | bounded | bounded, per item |
76
+ | Execution | one stage finishes before the next starts | continuous | ad hoc | pipelined, bounded buffers *(planned, phase 2)* |
77
+ | Resume after crash | per task / per run | cluster checkpoints | hand-rolled or none | per item *(planned, phase 3)* |
78
+ | Shared resources (browser, LLM client) | per process, awkward | n/a | manual | first-class *(planned, phase 4)* |
79
+ | Infra | server + database | cluster | none | none |
80
+ | Sweet spot | multi-team, multi-machine DAGs | 24/7 event streams | one-off | one node, 10³–10⁶ items, crash-prone I/O *(unmeasured)* |
81
+
82
+ The architectural difference tinyorch is built around: in a stage-at-a-time
83
+ orchestrator, data crossing a task boundary is materialized (in RAM or an
84
+ object store) before the next stage starts. tinyorch is designed to instead
85
+ pipeline items through stages with bounded buffers and checkpoint each item,
86
+ so that a rerun only does what is left and memory stays flat regardless of
87
+ input size. **This is a design goal, not a measured result**: none of it is
88
+ implemented in 0.1, and no benchmark backs the memory or resume claims yet.
89
+
90
+ ![Stage-at-a-time orchestrators vs tinyorch pipelining](https://raw.githubusercontent.com/murilegus/tinyorch/main/docs/assets/stage-vs-pipelined.svg)
91
+
92
+ *Conceptual diagram of the planned execution model, not a benchmark.*
93
+
94
+ **tinyorch is not a stream processor.** It is meant to take a *bounded* input
95
+ and execute it with streaming semantics. There are no windows, no event time, no
96
+ unbounded sources, no distributed state. If you need those, use Flink.
97
+
98
+ **It is not a replacement for Prefect or Airflow either.** When the pipeline
99
+ outgrows one machine or one team, those are the tools to move to; tinyorch
100
+ aims to make that boundary obvious rather than blurry.
101
+
102
+
103
+ ## 🏗️ Architecture
104
+
105
+ **Planned architecture** — today only the `@task` / `@flow` core exists; no
106
+ Protocols or adapters have been written yet. The intent: the core is
107
+ stdlib-only and defines Protocols; everything that touches the outside world
108
+ (state stores, browsers, LLM clients, CLI) is an adapter shipped as an
109
+ optional extra.
110
+
111
+ ![tinyorch architecture](https://raw.githubusercontent.com/murilegus/tinyorch/main/docs/assets/architecture.svg)
112
+
113
+
114
+ ## 🚫 Non-goals
115
+
116
+ - **Multi-node scheduling** or a long-running scheduler service — one process,
117
+ one machine, started by cron, a Cloud Run Job or your own code
118
+ - **Unbounded / event-time stream processing** — input is finite; no windows,
119
+ watermarks or distributed state
120
+ - **A full web UI** with authentication and multi-tenancy — a local, read-only
121
+ run viewer is planned as an optional extra
122
+ - **Time-based backfills** in the Airflow sense — runs are keyed by input, not
123
+ by schedule interval
124
+
125
+
126
+ ## 🧭 Roadmap
127
+
128
+ | Phase | Scope | Status |
129
+ |---|---|---|
130
+ | 1 | `@task` / `@flow`, run tracking, typed public API | ✅ 0.1 |
131
+ | 2 | Pipelining: streaming tasks, bounded buffers, concurrent execution | 🚧 next |
132
+ | 3 | Per-item checkpoints, resumable runs, pluggable state store | planned |
133
+ | 4 | Resources with lifecycle and injection; process workers for CPU-bound tasks | planned |
134
+ | 5 | Dogfooding on a real crawl + LLM pipeline; CLI | planned |
135
+
136
+ Progress is tracked in [GitHub milestones](https://github.com/murilegus/tinyorch/milestones).
137
+
138
+
139
+ ## 📄 Documentation
140
+
141
+ - [`docs/design/`](https://github.com/murilegus/tinyorch/tree/main/docs/design) — design decisions are recorded as ADRs.
142
+ ## 🎫 License
143
+
144
+ This project is licensed under the MIT License - see the [LICENSE](https://github.com/murilegus/tinyorch/blob/main/LICENSE) file for details.
@@ -1,6 +1,6 @@
1
1
  [project]
2
2
  name = "tinyorch"
3
- version = "0.1.0"
3
+ version = "0.1.2"
4
4
  description = "A tiny, zero-dependency orchestrator for single-node, item-level pipelines."
5
5
  readme = "README.md"
6
6
  license = "MIT"
@@ -11,6 +11,7 @@ classifiers = [
11
11
  "Programming Language :: Python :: 3",
12
12
  "Programming Language :: Python :: 3.12",
13
13
  "Programming Language :: Python :: 3.13",
14
+ "Programming Language :: Python :: 3.14",
14
15
  "Typing :: Typed",
15
16
  ]
16
17
  dependencies = []
@@ -44,6 +45,7 @@ include = [
44
45
  ]
45
46
  typeCheckingMode = "strict"
46
47
  failOnWarnings = true
48
+ reportUnnecessaryTypeIgnoreComment = "error"
47
49
 
48
50
  [[tool.basedpyright.executionEnvironments]]
49
51
  root = "tests"
@@ -51,7 +53,7 @@ root = "tests"
51
53
  [tool.pytest.ini_options]
52
54
  asyncio_mode = "auto"
53
55
  testpaths = ["tests"]
54
- addopts = "--cov=tinyorch --cov-report=term-missing --cov-fail-under=95"
56
+ addopts = "--cov=tinyorch --cov-report=term-missing"
55
57
 
56
58
  [tool.coverage.run]
57
59
  branch = true
@@ -61,7 +63,7 @@ source = ["tinyorch"]
61
63
  exclude_also = [
62
64
  "if TYPE_CHECKING:",
63
65
  "@overload",
64
- '\.\.\.',
66
+ '^\s*\.\.\.$',
65
67
  ]
66
68
 
67
69
  [tool.ruff]
@@ -1,6 +1,6 @@
1
1
  [project]
2
2
  name = "tinyorch"
3
- version = "0.1.0"
3
+ version = "0.1.2"
4
4
  description = "A tiny, zero-dependency orchestrator for single-node, item-level pipelines."
5
5
  readme = "README.md"
6
6
  authors = [{ name = "Murilegus", email = "murilegus@outlook.com" }]
@@ -12,6 +12,7 @@ classifiers = [
12
12
  "Programming Language :: Python :: 3",
13
13
  "Programming Language :: Python :: 3.12",
14
14
  "Programming Language :: Python :: 3.13",
15
+ "Programming Language :: Python :: 3.14",
15
16
  "Typing :: Typed",
16
17
  ]
17
18
  dependencies = []
@@ -38,6 +39,9 @@ build-backend = "uv_build"
38
39
  include = ["src", "tests"]
39
40
  typeCheckingMode = "strict"
40
41
  failOnWarnings = true
42
+ # Off by default even in strict; without it a stale `# pyright: ignore[...]`
43
+ # passes silently and the negative checks in tests/test_typing.py guard nothing.
44
+ reportUnnecessaryTypeIgnoreComment = "error"
41
45
 
42
46
  [[tool.basedpyright.executionEnvironments]]
43
47
  root = "tests"
@@ -45,7 +49,9 @@ root = "tests"
45
49
  [tool.pytest.ini_options]
46
50
  asyncio_mode = "auto"
47
51
  testpaths = ["tests"]
48
- addopts = "--cov=tinyorch --cov-report=term-missing --cov-fail-under=95"
52
+ # The coverage floor is enforced in CI (`pytest --cov-fail-under=95`), not here,
53
+ # so running a single test file locally does not fail on partial coverage.
54
+ addopts = "--cov=tinyorch --cov-report=term-missing"
49
55
 
50
56
  [tool.coverage.run]
51
57
  branch = true
@@ -55,7 +61,7 @@ source = ["tinyorch"]
55
61
  exclude_also = [
56
62
  "if TYPE_CHECKING:",
57
63
  "@overload",
58
- "\\.\\.\\.",
64
+ "^\\s*\\.\\.\\.$",
59
65
  ]
60
66
 
61
67
  [tool.ruff]
@@ -7,6 +7,7 @@ import inspect
7
7
  from typing import TYPE_CHECKING, cast, override
8
8
 
9
9
  from tinyorch._context import enter_run
10
+ from tinyorch._validate import check_wrappable
10
11
 
11
12
  if TYPE_CHECKING:
12
13
  from collections.abc import Awaitable, Callable
@@ -14,6 +15,7 @@ if TYPE_CHECKING:
14
15
 
15
16
  class Flow[**P, R]:
16
17
  def __init__(self, fn: Callable[P, R], *, name: str | None = None) -> None:
18
+ check_wrappable(fn, "flow")
17
19
  self.fn: Callable[P, R] = fn
18
20
  self.name: str = name or fn.__name__
19
21
  self.is_async: bool = inspect.iscoroutinefunction(fn)
@@ -8,6 +8,7 @@ import time
8
8
  from typing import TYPE_CHECKING, cast, override
9
9
 
10
10
  from tinyorch._context import RunContext, TaskCall, current_run
11
+ from tinyorch._validate import check_wrappable
11
12
 
12
13
  if TYPE_CHECKING:
13
14
  from collections.abc import Awaitable, Callable
@@ -15,6 +16,7 @@ if TYPE_CHECKING:
15
16
 
16
17
  class Task[**P, R]:
17
18
  def __init__(self, fn: Callable[P, R], *, name: str | None = None) -> None:
19
+ check_wrappable(fn, "task")
18
20
  self.fn: Callable[P, R] = fn
19
21
  self.name: str = name or fn.__name__
20
22
  self.is_async: bool = inspect.iscoroutinefunction(fn)
@@ -0,0 +1,32 @@
1
+ # SPDX-FileCopyrightText: © 2026 Murilegus <murilegus@outlook.com>
2
+ # SPDX-License-Identifier: MIT
3
+
4
+ from __future__ import annotations
5
+
6
+ import inspect
7
+ from typing import TYPE_CHECKING
8
+
9
+ if TYPE_CHECKING:
10
+ from collections.abc import Callable
11
+
12
+
13
+ def check_wrappable(fn: Callable[..., object], kind: str) -> None:
14
+ """Reject callables whose body would not run inside the bookkeeping.
15
+
16
+ Only plain ``def`` / ``async def`` functions and bound methods are
17
+ accepted. See ADR-002.
18
+ """
19
+ if not (inspect.isfunction(fn) or inspect.ismethod(fn)):
20
+ msg = (
21
+ f"@{kind} expects a function defined with 'def' or 'async def', "
22
+ f"got {type(fn).__name__}; wrap it in a function first"
23
+ )
24
+ raise TypeError(msg)
25
+
26
+ if inspect.isgeneratorfunction(fn) or inspect.isasyncgenfunction(fn):
27
+ msg = (
28
+ f"@{kind} does not support generator functions ({fn.__qualname__}): "
29
+ f"the body would run after the {kind} call has already returned. "
30
+ "Return a value from a plain 'def' or 'async def' instead"
31
+ )
32
+ raise TypeError(msg)
tinyorch-0.1.0/README.md DELETED
@@ -1,118 +0,0 @@
1
- # 🔀 tinyorch
2
-
3
- [![ci](https://github.com/murilegus/tinyorch/actions/workflows/ci.yml/badge.svg)](https://github.com/murilegus/tinyorch/actions/workflows/ci.yml)
4
- [![PyPI](https://img.shields.io/pypi/v/tinyorch)](https://pypi.org/project/tinyorch/)
5
- [![Python](https://img.shields.io/badge/python-3.12%20%7C%203.13-blue)](pyproject.toml)
6
- [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
7
-
8
- A tiny, zero-dependency orchestrator for single-node, item-level pipelines.
9
-
10
- 🚧 **Status**: Phase 1 complete — `@task` / `@flow` run sequentially with run
11
- tracking. No DAG, parallelism, retries or checkpoints yet; those are phase 2–3.
12
-
13
-
14
- ## 🚀 Quickstart
15
-
16
- ```bash
17
- pip install tinyorch
18
- ```
19
-
20
- ```python
21
- import asyncio
22
- from tinyorch import task, flow, current_run
23
-
24
-
25
- @task
26
- async def fetch(url: str) -> str:
27
- await asyncio.sleep(0.1) # pretend to hit the network
28
- return f"<html>{url}</html>"
29
-
30
-
31
- @task
32
- def parse(html: str) -> list[str]:
33
- return html.split()
34
-
35
-
36
- @flow
37
- async def crawl(urls: list[str]) -> int:
38
- for url in urls:
39
- parse(await fetch(url))
40
- run = current_run()
41
- return len(run.calls) if run else 0
42
-
43
-
44
- print(asyncio.run(crawl(["a", "b", "c"]))) # 6 task calls recorded
45
- ```
46
-
47
- A `@task` is a plain callable with metadata: outside a flow it just runs.
48
- Inside a `@flow`, every call is recorded on the current `RunContext`
49
- (`current_run()`), which also tracks the flow's own timing and failure.
50
- Sync and async functions both work; the wrapped signature is preserved for
51
- type checkers.
52
-
53
-
54
- ## 🤔 Why tinyorch?
55
-
56
- tinyorch is for the job that is **too small for Prefect, too finite for Flink,
57
- and too painful as a bare script**: a few thousand to a few million items
58
- (URLs, documents, records) flowing through I/O-heavy stages on one machine,
59
- where any stage can fail halfway.
60
-
61
- Three families of tools already exist for pipelines. Each is right for its
62
- sweet spot and awkward outside it:
63
-
64
- | | Workflow orchestrators<br>Airflow · Prefect · Dagster | Stream processors<br>Flink · Bytewax · Faust | Bare script<br>`asyncio.gather` + `tenacity` | **tinyorch** |
65
- |---|---|---|---|---|
66
- | Input | bounded, per task | **unbounded**, per event | bounded | bounded, per item |
67
- | Execution | one stage finishes before the next starts | continuous | ad hoc | pipelined, bounded buffers |
68
- | Resume after crash | per task / per run | cluster checkpoints | hand-rolled or none | per item |
69
- | Shared resources (browser, LLM client) | per process, awkward | n/a | manual | first-class |
70
- | Infra | server + database | cluster | none | none |
71
- | Sweet spot | multi-team, multi-machine DAGs | 24/7 event streams | one-off | one node, 10³–10⁶ items, crash-prone I/O |
72
-
73
- The architectural difference tinyorch is built around: in a stage-at-a-time
74
- orchestrator, data crossing a task boundary is materialized (in RAM or an
75
- object store) before the next stage starts. tinyorch instead pipelines items
76
- through stages with bounded buffers and checkpoints each item, so a rerun only
77
- does what is left and memory stays flat regardless of input size.
78
-
79
- ![Stage-at-a-time orchestrators vs tinyorch pipelining](docs/assets/stage-vs-pipelined.svg)
80
-
81
- **tinyorch is not a stream processor.** It takes a *bounded* input and
82
- executes it with streaming semantics. There are no windows, no event time, no
83
- unbounded sources, no distributed state. If you need those, use Flink.
84
-
85
- **It is not a replacement for Prefect or Airflow either.** When the pipeline
86
- outgrows one machine or one team, those are the tools to move to; tinyorch
87
- aims to make that boundary obvious rather than blurry.
88
-
89
-
90
- ## 🏗️ Architecture
91
-
92
- The core is stdlib-only and defines Protocols; everything that touches the
93
- outside world (state stores, browsers, LLM clients, CLI) is an adapter shipped
94
- as an optional extra.
95
-
96
- ![tinyorch architecture](docs/assets/architecture.svg)
97
-
98
-
99
- ## 🚫 Non-goals
100
-
101
- - **Multi-node scheduling** or a long-running scheduler service — one process,
102
- one machine, started by cron, a Cloud Run Job or your own code
103
- - **Unbounded / event-time stream processing** — input is finite; no windows,
104
- watermarks or distributed state
105
- - **A full web UI** with authentication and multi-tenancy — a local, read-only
106
- run viewer is planned as an optional extra
107
- - **Time-based backfills** in the Airflow sense — runs are keyed by input, not
108
- by schedule interval
109
-
110
-
111
- ## 📄 Documentation
112
-
113
- - [`docs/design/`](docs/design/) — design decisions are recorded as ADRs.
114
-
115
-
116
- ## 🎫 License
117
-
118
- This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
File without changes
File without changes
File without changes