celestialflow 3.1.8__tar.gz → 3.2.0__tar.gz

{celestialflow-3.1.8 → celestialflow-3.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: celestialflow
-Version: 3.1.8
+Version: 3.2.0
 Summary: A flexible GRAPH-based task orchestration framework.
 Author-email: Mr-xiaotian <mingxiaomingtian@gmail.com>
 License: MIT
@@ -62,10 +62,7 @@ TaskExecutor 实现了对任务的结果缓存，任务去重，进度条显示
 
 但除去直接使用 TaskExecutor，更重要的是使用其子类**TaskStage**。TaskStage 可以互相连接，形成具有上游与下游依赖关系的任务图（**TaskGraph**）。下游 stage 会自动接收上游执行完成的结果作为输入，从而形成明确的数据流。
 
-TaskStage 的任务执行模式只有两种：
-
-* **线性（serial）**
-* **多线程（thread）**
+TaskStage 的任务执行模式同样包含三种，与TaskExecutor中一致。
 
 在图级别上，每个 Stage 支持三种上下文模式：
 
@@ -153,15 +150,12 @@ def square(x):
 
 if __name__ == "__main__":
     # 定义两个任务节点
-    stage1 = TaskStage(add, execution_mode="thread", unpack_task_args=True)
-    stage2 = TaskStage(square, execution_mode="thread")
+    stage1 = TaskStage(name="Adder", func=add, execution_mode="thread", unpack_task_args=True, stage_mode="process")
+    stage2 = TaskStage(name="Squarer"func=square, execution_mode="thread", stage_mode="process")
 
     # 构建任务图结构
-    stage1 = TaskStage(func=add, execution_mode="thread", unpack_task_args=True, stage_mode="process", name="Adder")
-    stage2 = TaskStage(func=square, execution_mode="thread", stage_mode="process", name="Squarer")
-
     graph = TaskGraph()
-    graph.set_stages(root_stages=[stage1], stages=[stage2])
+    graph.set_stages(stages=[stage1, stage2])
     graph.connect([stage1], [stage2])
 
     # 初始化任务并启动
@@ -253,12 +247,12 @@ flowchart TD
 | 依赖包           | 说明 |
 | ----------------- | ---- |
 | **Python ≥ 3.10**  | 运行环境，建议使用 3.10 及以上版本 |
-| **tqdm**          | 控制台进度条显示，用于任务执行可视化 |
 | **fastapi**       | Web 服务接口框架（用于任务可视化与远程控制） |
 | **uvicorn**       | FastAPI 的高性能 ASGI 服务器 |
 | **requests**      | HTTP 客户端库，用于任务状态上报与远程调用 |
 | **networkx**      | 任务图（TaskGraph）结构与依赖分析 |
 | **jinja2**        | FastAPI 模板引擎，用于 Web 可视化界面渲染 |
+| **tqdm**          | 可选组件，进度条显示，用于任务执行可视化 |
 | **redis**         | 可选组件，用于分布式任务通信（`TaskRedis*` 系列模块） |
 | **celestialtree** | 可选组件，用于任务状态上报与远程调用（`ctree_client`） |
 
@@ -267,32 +261,31 @@ flowchart TD
 <p align="center">
   <img src="https://raw.githubusercontent.com/Mr-xiaotian/CelestialFlow/main/img/file_structure.svg" alt="FileStructure" />
   <br/>
-  <em>celestial-flow 3.1.8</em>
+  <em>celestial-flow 3.2.0</em>
 </p>
 
 (该视图由我的另一个项目[CelestialVault](https://github.com/Mr-xiaotian/CelestialVault)中inst_file.FileTree.print_tree()生成。转换为图片则借助[Carbon](https://carbon.now.sh)。)
 
 ## 版本日志（Version Log）
-- 3.1.8
+- 3.2.0
   - feat:
-    - [Important] 为stage_mode新增"thread"模式, 适合I/O密集型任务和不可pickle的函数;
-      - 在io密集任务的bench测试中, "thread-thread"相比最慢的"serial-serial"快8.7x, 并且优于之前最快的"process-thread";
-    - 新增docs/.../bench与docs/.../demo文档, 补充bench实测结果;
+    - [Important] 彻底废弃 `stage_mode="process"`, 移除所有 multiprocessing 依赖(MPValue, MPQueue, multiprocessing.Process);
+      - bench_graph_mode 数据表明 process 模式在所有场景下均慢于 thread 模式, 且引入大量序列化开销和 pickle 限制;
+    - [Important] 删除原有set_stages中手动输入的`root_stages`参数, 取而代之为通过scc缩合图计算出的一组`source_stages`
+      - 重补了不少图论课
+    - 将graph/stage/executor的默认log level从`SUCCESS`改为`INFO`, 也就是默认只显示开启关闭信息与错误
+    - 在web页面中添加配置按钮
+      - 当前仅支持设置刷新间隔与历史长度, 之后可以进行更多设置
   - refactor:
-    - [Important] 删除stage.set_stage_context, 改用graph.connect;
-      - 接口破坏性重构, 需要修改大量graph相关代码;
-    - [Important] name作为第一位参数在stage/executor中强制设定;
-      - 接口破坏性重构
-    - 优化async的executor_mode, 使其同时获取与消费任务, 而非集中消费;
-    - 为上版本retry机制的重构收尾;
+    - 由于stage_mode中取消`process`, 框架中部分为了适配`process`而进行的设计进行删除或者重构  
+      - 例如将所有的MPValue和MPQueue改为int与Queue
+      - 尚未进行严格的bench测试, 但应该会带来一定的性能优化
+    - 重构networkx图的建立过程, 现在直接通过节点与出边进行建立, 不再依赖递归
+    - 在重试检测机制中计算bytes类型的hash, 而非原本的string类型
+      - 根据 bench_hash_memory, 节省内存约23%
+    - 将节点状态中的deltas数据放在web端由js计算, 减少不必要的通信数据
   - fix:
-    - 修复find_unpickleable对bound method的误报;
-    - 修复stage重复检测错误;
-    - 修复retry任务无法被去重的问题;
-  - chore:
-    - 将demo/中所有test_函数名改为demo_函数名;
-    - 重组docs目录结构, 将reference改为src;
-    - docs中新增en/与ja/, 并将原有中文文档移至zh-CN/;
+    - 删除InQueue.get中的错误捕获, 这会导致错过panic级error
 
 更多过往日志可看:
 

{celestialflow-3.1.8 → celestialflow-3.2.0}/README.md

@@ -37,10 +37,7 @@ TaskExecutor 实现了对任务的结果缓存，任务去重，进度条显示
 
 但除去直接使用 TaskExecutor，更重要的是使用其子类**TaskStage**。TaskStage 可以互相连接，形成具有上游与下游依赖关系的任务图（**TaskGraph**）。下游 stage 会自动接收上游执行完成的结果作为输入，从而形成明确的数据流。
 
-TaskStage 的任务执行模式只有两种：
-
-* **线性（serial）**
-* **多线程（thread）**
+TaskStage 的任务执行模式同样包含三种，与TaskExecutor中一致。
 
 在图级别上，每个 Stage 支持三种上下文模式：
 
@@ -128,15 +125,12 @@ def square(x):
 
 if __name__ == "__main__":
     # 定义两个任务节点
-    stage1 = TaskStage(add, execution_mode="thread", unpack_task_args=True)
-    stage2 = TaskStage(square, execution_mode="thread")
+    stage1 = TaskStage(name="Adder", func=add, execution_mode="thread", unpack_task_args=True, stage_mode="process")
+    stage2 = TaskStage(name="Squarer"func=square, execution_mode="thread", stage_mode="process")
 
     # 构建任务图结构
-    stage1 = TaskStage(func=add, execution_mode="thread", unpack_task_args=True, stage_mode="process", name="Adder")
-    stage2 = TaskStage(func=square, execution_mode="thread", stage_mode="process", name="Squarer")
-
     graph = TaskGraph()
-    graph.set_stages(root_stages=[stage1], stages=[stage2])
+    graph.set_stages(stages=[stage1, stage2])
     graph.connect([stage1], [stage2])
 
     # 初始化任务并启动
@@ -228,12 +222,12 @@ flowchart TD
 | 依赖包           | 说明 |
 | ----------------- | ---- |
 | **Python ≥ 3.10**  | 运行环境，建议使用 3.10 及以上版本 |
-| **tqdm**          | 控制台进度条显示，用于任务执行可视化 |
 | **fastapi**       | Web 服务接口框架（用于任务可视化与远程控制） |
 | **uvicorn**       | FastAPI 的高性能 ASGI 服务器 |
 | **requests**      | HTTP 客户端库，用于任务状态上报与远程调用 |
 | **networkx**      | 任务图（TaskGraph）结构与依赖分析 |
 | **jinja2**        | FastAPI 模板引擎，用于 Web 可视化界面渲染 |
+| **tqdm**          | 可选组件，进度条显示，用于任务执行可视化 |
 | **redis**         | 可选组件，用于分布式任务通信（`TaskRedis*` 系列模块） |
 | **celestialtree** | 可选组件，用于任务状态上报与远程调用（`ctree_client`） |
 
@@ -242,32 +236,31 @@ flowchart TD
 <p align="center">
   <img src="https://raw.githubusercontent.com/Mr-xiaotian/CelestialFlow/main/img/file_structure.svg" alt="FileStructure" />
   <br/>
-  <em>celestial-flow 3.1.8</em>
+  <em>celestial-flow 3.2.0</em>
 </p>
 
 (该视图由我的另一个项目[CelestialVault](https://github.com/Mr-xiaotian/CelestialVault)中inst_file.FileTree.print_tree()生成。转换为图片则借助[Carbon](https://carbon.now.sh)。)
 
 ## 版本日志（Version Log）
-- 3.1.8
+- 3.2.0
   - feat:
-    - [Important] 为stage_mode新增"thread"模式, 适合I/O密集型任务和不可pickle的函数;
-      - 在io密集任务的bench测试中, "thread-thread"相比最慢的"serial-serial"快8.7x, 并且优于之前最快的"process-thread";
-    - 新增docs/.../bench与docs/.../demo文档, 补充bench实测结果;
+    - [Important] 彻底废弃 `stage_mode="process"`, 移除所有 multiprocessing 依赖(MPValue, MPQueue, multiprocessing.Process);
+      - bench_graph_mode 数据表明 process 模式在所有场景下均慢于 thread 模式, 且引入大量序列化开销和 pickle 限制;
+    - [Important] 删除原有set_stages中手动输入的`root_stages`参数, 取而代之为通过scc缩合图计算出的一组`source_stages`
+      - 重补了不少图论课
+    - 将graph/stage/executor的默认log level从`SUCCESS`改为`INFO`, 也就是默认只显示开启关闭信息与错误
+    - 在web页面中添加配置按钮
+      - 当前仅支持设置刷新间隔与历史长度, 之后可以进行更多设置
   - refactor:
-    - [Important] 删除stage.set_stage_context, 改用graph.connect;
-      - 接口破坏性重构, 需要修改大量graph相关代码;
-    - [Important] name作为第一位参数在stage/executor中强制设定;
-      - 接口破坏性重构
-    - 优化async的executor_mode, 使其同时获取与消费任务, 而非集中消费;
-    - 为上版本retry机制的重构收尾;
+    - 由于stage_mode中取消`process`, 框架中部分为了适配`process`而进行的设计进行删除或者重构  
+      - 例如将所有的MPValue和MPQueue改为int与Queue
+      - 尚未进行严格的bench测试, 但应该会带来一定的性能优化
+    - 重构networkx图的建立过程, 现在直接通过节点与出边进行建立, 不再依赖递归
+    - 在重试检测机制中计算bytes类型的hash, 而非原本的string类型
+      - 根据 bench_hash_memory, 节省内存约23%
+    - 将节点状态中的deltas数据放在web端由js计算, 减少不必要的通信数据
   - fix:
-    - 修复find_unpickleable对bound method的误报;
-    - 修复stage重复检测错误;
-    - 修复retry任务无法被去重的问题;
-  - chore:
-    - 将demo/中所有test_函数名改为demo_函数名;
-    - 重组docs目录结构, 将reference改为src;
-    - docs中新增en/与ja/, 并将原有中文文档移至zh-CN/;
+    - 删除InQueue.get中的错误捕获, 这会导致错过panic级error
 
 更多过往日志可看:
 

{celestialflow-3.1.8 → celestialflow-3.2.0}/pyproject.toml

@@ -1,81 +1,80 @@
-[build-system]
-requires = ["setuptools>=61.0", "wheel"]
-build-backend = "setuptools.build_meta"
-
-[project]
-name = "celestialflow"                   
-version = "3.1.8"
-description = "A flexible GRAPH-based task orchestration framework."
-readme = "README.md"               
-license = { text = "MIT" }    
-authors = [{ name = "Mr-xiaotian", email = "mingxiaomingtian@gmail.com" }]
-keywords = ["workflow", "task", "graph", "async", "CelestialFlow"]
-requires-python = ">=3.10"
-
-dependencies = [
-    "tqdm",
-    "fastapi",
-    "uvicorn",
-    "requests",
-    "networkx",
-    "redis",
-    "jinja2",
-    "celestialtree>=0.1.2",
-]
-
-classifiers = [
-    "Programming Language :: Python :: 3",
-    "License :: OSI Approved :: MIT License",
-    "Operating System :: OS Independent",
-    "Framework :: FastAPI",
-    "Topic :: Software Development :: Libraries",
-]
-
-[project.urls]
-"Homepage" = "https://github.com/Mr-xiaotian/CelestialFlow"
-"Bug Tracker" = "https://github.com/Mr-xiaotian/CelestialFlow/issues"
-
-[project.scripts]
-celestialflow-web = "celestialflow.web.core_server:main_entry"
-
-[tool.setuptools]
-license-files = []
-
-[tool.setuptools.package-data]
-celestialflow = [
-    "web/templates/**/*.html",
-    "web/static/**/*",
-    "web/config.json"
-]
-
-[tool.pytest.ini_options]
-addopts = "-s"
-log_cli = true
-log_cli_level = "INFO"
-log_cli_format = "%(asctime)s [%(levelname)s] %(message)s"
-log_cli_date_format = "%Y-%m-%d %H:%M:%S"
-
-filterwarnings = [
-    "ignore::DeprecationWarning",
-    "ignore::pytest.PytestDeprecationWarning",
-]
-
-asyncio_default_fixture_loop_scope = "function"
-
-[dependency-groups]
-dev = [
-    "build>=1.2.2.post1",
-    "twine>=6.1.0",
-    "pytest>=8.3.4",
-    "pytest-asyncio>=0.25.3",
-    "httpx>=0.28.1",
-    "python-dotenv>=1.2.2",
-    "dill",
-    "mypy>=1.19.1",
-    "ruff",
-]
-
-[tool.mypy]
-python_version = "3.10"
-no_implicit_optional = false
-disable_error_code = ["import-untyped"]
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "celestialflow"                   
+version = "3.2.0"
+description = "A flexible GRAPH-based task orchestration framework."
+readme = "README.md"
+license = { text = "MIT" }
+authors = [{ name = "Mr-xiaotian", email = "mingxiaomingtian@gmail.com" }]
+keywords = ["workflow", "task", "graph", "async", "CelestialFlow"]
+requires-python = ">=3.10"
+
+dependencies = [
+    "tqdm",
+    "fastapi",
+    "uvicorn",
+    "requests",
+    "networkx",
+    "redis",
+    "jinja2",
+    "celestialtree>=0.1.2",
+]
+
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Framework :: FastAPI",
+    "Topic :: Software Development :: Libraries",
+]
+
+[project.urls]
+"Homepage" = "https://github.com/Mr-xiaotian/CelestialFlow"
+"Bug Tracker" = "https://github.com/Mr-xiaotian/CelestialFlow/issues"
+
+[project.scripts]
+celestialflow-web = "celestialflow.web.core_server:main_entry"
+
+[tool.setuptools]
+license-files = []
+
+[tool.setuptools.package-data]
+celestialflow = [
+    "web/templates/**/*.html",
+    "web/static/**/*",
+    "web/config.json"
+]
+
+[tool.pytest.ini_options]
+addopts = "-s"
+log_cli = true
+log_cli_level = "INFO"
+log_cli_format = "%(asctime)s [%(levelname)s] %(message)s"
+log_cli_date_format = "%Y-%m-%d %H:%M:%S"
+
+filterwarnings = [
+    "ignore::DeprecationWarning",
+    "ignore::pytest.PytestDeprecationWarning",
+]
+
+asyncio_default_fixture_loop_scope = "function"
+
+[dependency-groups]
+dev = [
+    "build>=1.2.2.post1",
+    "twine>=6.1.0",
+    "pytest>=8.3.4",
+    "pytest-asyncio>=0.25.3",
+    "httpx>=0.28.1",
+    "python-dotenv>=1.2.2",
+    "ruff>=0.15.0",
+    "pyright>=1.1.409",
+]
+
+[tool.pyright]
+include = ["src"]
+strict = ["src/celestialflow/runtime", "src/celestialflow/stage", "src/celestialflow/graph", "src/celestialflow/persistence", "src/celestialflow/funnel", "src/celestialflow/observability", "src/celestialflow/utils"]
+stubsPath = "typings"

{celestialflow-3.1.8 → celestialflow-3.2.0}/src/celestialflow/__init__.py

@@ -8,6 +8,7 @@ from .graph import (
     TaskLoop,
     TaskWheel,
 )
+from .observability import BaseObserver, CallbackObserver, TaskProgress
 from .persistence.util_jsonl import (
     load_jsonl_logs,
     load_task_by_error,
@@ -26,7 +27,7 @@ from .stage import (
 )
 from .utils.util_benchmark import benchmark_executor, benchmark_graph
 from .utils.util_format import format_table
-from .web.core_server import TaskWebServer
+from .web import TaskWebServer
 
 __all__ = [
     "TaskGraph",
@@ -36,6 +37,9 @@ __all__ = [
     "TaskComplete",
     "TaskWheel",
     "TaskGrid",
+    "BaseObserver",
+    "CallbackObserver",
+    "TaskProgress",
     "TaskExecutor",
     "TaskStage",
     "TaskSplitter",

{celestialflow-3.1.8 → celestialflow-3.2.0}/src/celestialflow/funnel/core_spout.py

@@ -1,12 +1,10 @@
 # funnel/core_spout.py
 from __future__ import annotations
 
-from multiprocessing import Queue as MPQueue
-from queue import Empty
+from queue import Empty, Queue
 from threading import Thread
 from typing import Any
 
-from ..runtime.util_queue import cleanup_mpqueue
 from ..runtime.util_types import TERMINATION_SIGNAL, TerminationSignal
 
 
@@ -15,7 +13,7 @@ class BaseSpout:
 
     def __init__(self) -> None:
         """初始化监听器及其内部队列和线程引用。"""
-        self.queue: Any = MPQueue()
+        self.queue: Queue[Any] = Queue()
         self._thread: Thread | None = None
 
     def _before_start(self) -> None:
@@ -46,7 +44,7 @@ class BaseSpout:
             except Exception:  # ← 新增：防止线程崩溃
                 continue  # 或记录到 stderr，至少不丢后续记录
 
-    def get_queue(self) -> MPQueue:
+    def get_queue(self) -> Queue[Any]:
         """获取监听器的输入队列。"""
         return self.queue
 
@@ -58,5 +56,4 @@ class BaseSpout:
         self.queue.put(TERMINATION_SIGNAL)
         self._thread.join()
         self._thread = None
-        cleanup_mpqueue(self.queue)
         self._after_stop()