celestialflow 3.1.5__tar.gz → 3.1.7__tar.gz

{celestialflow-3.1.5 → celestialflow-3.1.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: celestialflow
-Version: 3.1.5
+Version: 3.1.7
 Summary: A flexible GRAPH-based task orchestration framework.
 Author-email: Mr-xiaotian <mingxiaomingtian@gmail.com>
 License: MIT
@@ -48,11 +48,10 @@ Requires-Dist: celestialtree>=0.1.2
 - 相比 Airflow/Dagster 更轻、更快开始
 - 相比 multiprocessing/threading 更结构化，可直接表达 loop / complete graph 等复杂依赖模式
 
-框架的基本单元为 **TaskExecutor**，可独立运行，并支持四种执行模式：
+框架的基本单元为 **TaskExecutor**，可独立运行，并支持三种执行模式：
 
 * **线性（serial）**
 * **多线程（thread）**
-* **多进程（process）**
 * **协程（async）**
 
 TaskExecutor 实现了对任务的结果缓存，任务去重，进度条显示，多执行模式比较等功能，单独使用也很好用。
@@ -133,7 +132,7 @@ flowchart LR
 uv pip install celestialflow
 
 # 不过也可以直接使用 `pip`
-uv pip install celestialflow
+pip install celestialflow
 ```
 
 一个简单的可运行代码:
@@ -175,16 +174,16 @@ if __name__ == "__main__":
 
 若你想了解框架的整体结构与核心组件，下面的参考文档会对你有帮助：
 
-- [stage/executor.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/executor.md)
-- [stage/stage.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/stage.md)
-- [graph/graph.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/graph.md)
-- [runtime/progress.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/progress.md)
-- [runtime/metrics.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/metrics.md)
-- [runtime/queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/queue.md)
-- [stage/nodes.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/nodes.md)
-- [observability/report.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/observability/report.md)
-- [graph/structure.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/structure.md)
-- [web/server.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/web/server.md)
+- [stage/core_executor.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/core_executor.md)
+- [stage/core_stage.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/core_stage.md)
+- [graph/core_graph.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/core_graph.md)
+- [observability/core_progress.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/observability/core_progress.md)
+- [runtime/core_metrics.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/core_metrics.md)
+- [runtime/core_queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/core_queue.md)
+- [stage/core_stages.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/core_stages.md)
+- [observability/core_report.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/observability/core_report.md)
+- [graph/core_structure.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/core_structure.md)
+- [web/core_server.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/web/core_server.md)
 - [other/go_worker.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/other/go_worker.md)
 
 推荐阅读顺序:
@@ -202,7 +201,7 @@ flowchart TD
     TM --> TME[TaskMetrics.md]
 
     TG --> TQ[TaskQueue.md]
-    TG --> TN[TaskNodes.md]
+    TG --> TN[TaskStages.md]
     TG --> TR[TaskReport.md]
     TG --> TSR[TaskStructure.md]
 
@@ -218,11 +217,11 @@ flowchart TD
 
 以下三篇可以作为补充阅读:
 
-- [runtime/tools.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/tools.md)
-- [runtime/types.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/types.md)
-- [runtime/errors.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/errors.md)
-- [persistence/fail.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/fail.md)
-- [persistence/log.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/log.md)
+- [runtime/util_queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/util_queue.md)
+- [runtime/util_types.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/util_types.md)
+- [runtime/util_errors.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/util_errors.md)
+- [persistence/core_fail.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/core_fail.md)
+- [persistence/core_log.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/core_log.md)
 
 如果你更喜欢通过完整案例理解框架的运行方式，可以参考这篇从零开始构建 TaskGraph 的教程：
 
@@ -234,7 +233,7 @@ flowchart TD
 
 ## 环境要求（Requirements）
 
-**CelestialFlow** 基于 Python 3.8+，并依赖以下核心组件。  
+**CelestialFlow** 基于 Python 3.10+，并依赖以下核心组件。  
 请确保你的环境能够正常安装这些依赖（`pip install celestialflow` 会自动安装）。
 
 | 依赖包           | 说明 |
@@ -254,22 +253,24 @@ flowchart TD
 <p align="center">
   <img src="https://raw.githubusercontent.com/Mr-xiaotian/CelestialFlow/main/img/file_structure.svg" alt="FileStructure" />
   <br/>
-  <em>celestial-flow 3.1.5</em>
+  <em>celestial-flow 3.1.7</em>
 </p>
 
 (该视图由我的另一个项目[CelestialVault](https://github.com/Mr-xiaotian/CelestialVault)中inst_file.FileTree.print_tree()生成。转换为图片则借助[Carbon](https://carbon.now.sh)。)
 
 ## 版本日志（Version Log）
-- 3.1.5
-  - feat
-    - 大幅修改节点状态卡片的色彩视觉设计. 包括: 取消原本的悬浮设计, 改为平面化设计; 让进度条更直接的显示不同任务完成状态的比例; 使用左边框来显示节点是否在运行中. 而非原本的badge;
-    - 前端的error数据拉取不再每次都全量拉取, 而是只拉取自己当前没有的数据;
-  - refactor
-    - 大幅重构前端代码中的色彩管理, 现在色彩的一致性更好;
-    - 修改前端代码中的数据更新判断, 现在相关判断交给serve.py;
-  - fix
-    - 修复节点已运行时间在节点完成后显示为0的问题
-    - 修复在最新fastapi版本下TemplateResponse参数改变导致的问题
+- 3.1.7
+  - feat:
+    - [Important] 删除executor中的"process"模式, 它跟新的retry机制实在难以匹配;
+  - refactor:
+    - [Important] 大幅重构retry机制, 现在不再让retry任务重进task_queue, 而是在worker中直接解决;
+      - 模仿CelestialForge中grow的做法;
+    - 大幅重构原本成功与失败结果的缓存机制, 并将原有get_success/error_dict改名为get_success/error_pairs, 以避免task无法作为键时会导致的问题; 
+      - fail数据从fail.jsonl中提取;
+      - success数据从result_queue中直接获取, 为此添加了SuccessSpout作为集合端;
+  - fix:
+    - 修复部分任务类型无法在log_spout中dump进jsonl文件的问题;
+    - 修复在executor运行时, retry添加任务total数的问题, 虽然这样很帅;
 
 ## Star 历史趋势（Star History）
 

{celestialflow-3.1.5 → celestialflow-3.1.7}/README.md

@@ -23,11 +23,10 @@
 - 相比 Airflow/Dagster 更轻、更快开始
 - 相比 multiprocessing/threading 更结构化，可直接表达 loop / complete graph 等复杂依赖模式
 
-框架的基本单元为 **TaskExecutor**，可独立运行，并支持四种执行模式：
+框架的基本单元为 **TaskExecutor**，可独立运行，并支持三种执行模式：
 
 * **线性（serial）**
 * **多线程（thread）**
-* **多进程（process）**
 * **协程（async）**
 
 TaskExecutor 实现了对任务的结果缓存，任务去重，进度条显示，多执行模式比较等功能，单独使用也很好用。
@@ -108,7 +107,7 @@ flowchart LR
 uv pip install celestialflow
 
 # 不过也可以直接使用 `pip`
-uv pip install celestialflow
+pip install celestialflow
 ```
 
 一个简单的可运行代码:
@@ -150,16 +149,16 @@ if __name__ == "__main__":
 
 若你想了解框架的整体结构与核心组件，下面的参考文档会对你有帮助：
 
-- [stage/executor.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/executor.md)
-- [stage/stage.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/stage.md)
-- [graph/graph.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/graph.md)
-- [runtime/progress.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/progress.md)
-- [runtime/metrics.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/metrics.md)
-- [runtime/queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/queue.md)
-- [stage/nodes.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/nodes.md)
-- [observability/report.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/observability/report.md)
-- [graph/structure.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/structure.md)
-- [web/server.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/web/server.md)
+- [stage/core_executor.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/core_executor.md)
+- [stage/core_stage.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/core_stage.md)
+- [graph/core_graph.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/core_graph.md)
+- [observability/core_progress.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/observability/core_progress.md)
+- [runtime/core_metrics.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/core_metrics.md)
+- [runtime/core_queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/core_queue.md)
+- [stage/core_stages.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/core_stages.md)
+- [observability/core_report.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/observability/core_report.md)
+- [graph/core_structure.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/core_structure.md)
+- [web/core_server.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/web/core_server.md)
 - [other/go_worker.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/other/go_worker.md)
 
 推荐阅读顺序:
@@ -177,7 +176,7 @@ flowchart TD
     TM --> TME[TaskMetrics.md]
 
     TG --> TQ[TaskQueue.md]
-    TG --> TN[TaskNodes.md]
+    TG --> TN[TaskStages.md]
     TG --> TR[TaskReport.md]
     TG --> TSR[TaskStructure.md]
 
@@ -193,11 +192,11 @@ flowchart TD
 
 以下三篇可以作为补充阅读:
 
-- [runtime/tools.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/tools.md)
-- [runtime/types.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/types.md)
-- [runtime/errors.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/errors.md)
-- [persistence/fail.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/fail.md)
-- [persistence/log.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/log.md)
+- [runtime/util_queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/util_queue.md)
+- [runtime/util_types.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/util_types.md)
+- [runtime/util_errors.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/util_errors.md)
+- [persistence/core_fail.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/core_fail.md)
+- [persistence/core_log.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/core_log.md)
 
 如果你更喜欢通过完整案例理解框架的运行方式，可以参考这篇从零开始构建 TaskGraph 的教程：
 
@@ -209,7 +208,7 @@ flowchart TD
 
 ## 环境要求（Requirements）
 
-**CelestialFlow** 基于 Python 3.8+，并依赖以下核心组件。  
+**CelestialFlow** 基于 Python 3.10+，并依赖以下核心组件。  
 请确保你的环境能够正常安装这些依赖（`pip install celestialflow` 会自动安装）。
 
 | 依赖包           | 说明 |
@@ -229,22 +228,24 @@ flowchart TD
 <p align="center">
   <img src="https://raw.githubusercontent.com/Mr-xiaotian/CelestialFlow/main/img/file_structure.svg" alt="FileStructure" />
   <br/>
-  <em>celestial-flow 3.1.5</em>
+  <em>celestial-flow 3.1.7</em>
 </p>
 
 (该视图由我的另一个项目[CelestialVault](https://github.com/Mr-xiaotian/CelestialVault)中inst_file.FileTree.print_tree()生成。转换为图片则借助[Carbon](https://carbon.now.sh)。)
 
 ## 版本日志（Version Log）
-- 3.1.5
-  - feat
-    - 大幅修改节点状态卡片的色彩视觉设计. 包括: 取消原本的悬浮设计, 改为平面化设计; 让进度条更直接的显示不同任务完成状态的比例; 使用左边框来显示节点是否在运行中. 而非原本的badge;
-    - 前端的error数据拉取不再每次都全量拉取, 而是只拉取自己当前没有的数据;
-  - refactor
-    - 大幅重构前端代码中的色彩管理, 现在色彩的一致性更好;
-    - 修改前端代码中的数据更新判断, 现在相关判断交给serve.py;
-  - fix
-    - 修复节点已运行时间在节点完成后显示为0的问题
-    - 修复在最新fastapi版本下TemplateResponse参数改变导致的问题
+- 3.1.7
+  - feat:
+    - [Important] 删除executor中的"process"模式, 它跟新的retry机制实在难以匹配;
+  - refactor:
+    - [Important] 大幅重构retry机制, 现在不再让retry任务重进task_queue, 而是在worker中直接解决;
+      - 模仿CelestialForge中grow的做法;
+    - 大幅重构原本成功与失败结果的缓存机制, 并将原有get_success/error_dict改名为get_success/error_pairs, 以避免task无法作为键时会导致的问题; 
+      - fail数据从fail.jsonl中提取;
+      - success数据从result_queue中直接获取, 为此添加了SuccessSpout作为集合端;
+  - fix:
+    - 修复部分任务类型无法在log_spout中dump进jsonl文件的问题;
+    - 修复在executor运行时, retry添加任务total数的问题, 虽然这样很帅;
 
 ## Star 历史趋势（Star History）
 

{celestialflow-3.1.5 → celestialflow-3.1.7}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "celestialflow"                   
-version = "3.1.5"
+version = "3.1.7"
 description = "A flexible GRAPH-based task orchestration framework."
 readme = "README.md"               
 license = { text = "MIT" }    
@@ -36,7 +36,7 @@ classifiers = [
 "Bug Tracker" = "https://github.com/Mr-xiaotian/CelestialFlow/issues"
 
 [project.scripts]
-celestialflow-web = "celestialflow.web.server:main_entry"
+celestialflow-web = "celestialflow.web.core_server:main_entry"
 
 [tool.setuptools]
 license-files = []
@@ -70,5 +70,12 @@ dev = [
     "pytest-asyncio>=0.25.3",
     "httpx>=0.28.1",
     "python-dotenv>=1.2.2",
-    "black>=26.1.0",
+    "dill",
+    "mypy>=1.19.1",
+    "ruff",
 ]
+
+[tool.mypy]
+python_version = "3.10"
+no_implicit_optional = false
+disable_error_code = ["import-untyped"]

{celestialflow-3.1.5 → celestialflow-3.1.7}/src/celestialflow/__init__.py

@@ -1,32 +1,32 @@
 # __init__.py
-from .stage import (
-    TaskExecutor,
-    TaskStage,
-    TaskSplitter,
-    TaskRedisTransport,
-    TaskRedisSource,
-    TaskRedisAck,
-    TaskRouter,
-)
 from .graph import (
-    TaskGraph,
     TaskChain,
-    TaskLoop,
-    TaskCross,
     TaskComplete,
-    TaskWheel,
+    TaskCross,
+    TaskGraph,
     TaskGrid,
+    TaskLoop,
+    TaskWheel,
 )
 from .persistence.util_jsonl import (
     load_jsonl_logs,
-    load_task_by_stage,
     load_task_by_error,
+    load_task_by_stage,
 )
-from .runtime.util_types import TerminationSignal
 from .runtime.util_hash import make_hashable
+from .runtime.util_types import TerminationSignal
+from .stage import (
+    TaskExecutor,
+    TaskRedisAck,
+    TaskRedisSource,
+    TaskRedisTransport,
+    TaskRouter,
+    TaskSplitter,
+    TaskStage,
+)
+from .utils.util_benchmark import benchmark_executor, benchmark_graph
 from .utils.util_format import format_table
-from .utils.util_benchmark import benchmark_graph, benchmark_executor
-from .web.server import TaskWebServer
+from .web.core_server import TaskWebServer
 
 __all__ = [
     "TaskGraph",

celestialflow-3.1.7/src/celestialflow/funnel/__init__.py

@@ -0,0 +1,8 @@
+# funnel/__init__.py
+from .core_spout import BaseSpout
+from .core_inlet import BaseInlet
+
+__all__ = [
+    "BaseSpout",
+    "BaseInlet",
+]

celestialflow-3.1.7/src/celestialflow/funnel/core_inlet.py

@@ -0,0 +1,12 @@
+# funnel/core_inlet.py
+from __future__ import annotations
+
+from typing import Any
+
+
+class BaseInlet:
+    def __init__(self, queue: Any) -> None:
+        self.queue: Any = queue
+
+    def _funnel(self, record: Any) -> None:
+        self.queue.put(record)

celestialflow-3.1.5/src/celestialflow/persistence/core_base.py → celestialflow-3.1.7/src/celestialflow/funnel/core_spout.py

@@ -1,23 +1,24 @@
-# persistence/core_base.py
+# funnel/core_spout.py
 from __future__ import annotations
 
 from multiprocessing import Queue as MPQueue
 from queue import Empty
 from threading import Thread
+from typing import Any
 
 from ..runtime.util_queue import cleanup_mpqueue
-from ..runtime.util_types import TerminationSignal, TERMINATION_SIGNAL
+from ..runtime.util_types import TERMINATION_SIGNAL, TerminationSignal
 
 
-class BaseListener:
+class BaseSpout:
     def __init__(self) -> None:
-        self.queue = MPQueue()
+        self.queue: Any = MPQueue()
         self._thread: Thread | None = None
 
     def _before_start(self) -> None:
         return None
 
-    def _handle_record(self, record) -> None:
+    def _handle_record(self, record: Any) -> None:
         raise NotImplementedError
 
     def _after_stop(self) -> None:
@@ -26,10 +27,10 @@ class BaseListener:
     def start(self) -> None:
         self._before_start()
         if self._thread is None or not self._thread.is_alive():
-            self._thread = Thread(target=self._listen, daemon=True)
+            self._thread = Thread(target=self._spout, daemon=True)
             self._thread.start()
 
-    def _listen(self) -> None:
+    def _spout(self) -> None:
         while True:
             try:
                 record = self.queue.get(timeout=0.5)
@@ -51,11 +52,3 @@ class BaseListener:
         self._thread = None
         cleanup_mpqueue(self.queue)
         self._after_stop()
-
-
-class BaseSinker:
-    def __init__(self, queue: MPQueue) -> None:
-        self.queue: MPQueue = queue
-
-    def _sink(self, record) -> None:
-        self.queue.put(record)

{celestialflow-3.1.5 → celestialflow-3.1.7}/src/celestialflow/graph/__init__.py

@@ -2,11 +2,11 @@
 from .core_graph import TaskGraph
 from .core_structure import (
     TaskChain,
-    TaskLoop,
-    TaskCross,
     TaskComplete,
-    TaskWheel,
+    TaskCross,
     TaskGrid,
+    TaskLoop,
+    TaskWheel,
 )
 
 __all__ = [