celestialflow 3.1.7__tar.gz → 3.1.8__tar.gz

{celestialflow-3.1.7 → celestialflow-3.1.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: celestialflow
-Version: 3.1.7
+Version: 3.1.8
 Summary: A flexible GRAPH-based task orchestration framework.
 Author-email: Mr-xiaotian <mingxiaomingtian@gmail.com>
 License: MIT
@@ -43,6 +43,10 @@ Requires-Dist: celestialtree>=0.1.2
   <img src="https://img.shields.io/badge/Distributed-Worker%20Friendly-orange">
 </p>
 
+<p align="center">
+  <a href="README.md">中文</a> | <a href="docs/en/README.md">English</a> | <a href="docs/ja/README.md">日本語</a>
+</p>
+
 **CelestialFlow** 是一个轻量级但功能完全的任务流框架，适合需要 **复杂依赖关系**、**灵活执行模型**、**跨设备运行**与**实时可视化监控** 的中/大型 Python 任务系统。
 
 - 相比 Airflow/Dagster 更轻、更快开始
@@ -63,9 +67,10 @@ TaskStage 的任务执行模式只有两种：
 * **线性（serial）**
 * **多线程（thread）**
 
-在图级别上，每个 Stage 支持两种上下文模式：
+在图级别上，每个 Stage 支持三种上下文模式：
 
 * **线性执行（serial layout）**：当前节点执行完毕再启动下一节点（下游节点可提前接收任务但不会立即执行）。
+* **线程执行（thread layout）**：当前节点在主进程的独立线程中启动，适合 I/O 密集型任务和不可 pickle 的函数（如 lambda）。
 * **并行执行（process layout）**：当前节点启动后立刻前去启动下一节点。
 
 TaskGraph 能构建完整的 **有向图结构（Directed Graph）**，不仅支持传统的有向无环图（DAG），也能灵活表达 **树形（Tree）**、**环形（loop）** 乃至于 **完全图（Complete Graph）** 形式的任务依赖。
@@ -152,9 +157,12 @@ if __name__ == "__main__":
     stage2 = TaskStage(square, execution_mode="thread")
 
     # 构建任务图结构
-    stage1.set_graph_context([stage2], stage_mode="process", stage_name="Adder")
-    stage2.set_graph_context([], stage_mode="process", stage_name="Squarer")
-    graph = TaskGraph([stage1])
+    stage1 = TaskStage(func=add, execution_mode="thread", unpack_task_args=True, stage_mode="process", name="Adder")
+    stage2 = TaskStage(func=square, execution_mode="thread", stage_mode="process", name="Squarer")
+
+    graph = TaskGraph()
+    graph.set_stages(root_stages=[stage1], stages=[stage2])
+    graph.connect([stage1], [stage2])
 
     # 初始化任务并启动
     graph.start_graph({stage1.get_tag(): [(1, 2), (3, 4), (5, 6)]})
@@ -162,29 +170,23 @@ if __name__ == "__main__":
 
 注意不要在.ipynb中运行。
 
-👉 想查看完整Quick Start，请见[Quick Start](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/quick_start.md)
+👉 想查看完整Quick Start，请见[Quick Start](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/quick_start.md)
 
 ## 深入阅读（Further Reading）
 
-(以下文档完善中)
-
-你可以继续运行更多的测试代码，这里记录了各个测试文件与其中的测试函数说明：
-
-[📄tests/README.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/tests/README.md)
-
 若你想了解框架的整体结构与核心组件，下面的参考文档会对你有帮助：
 
-- [stage/core_executor.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/core_executor.md)
-- [stage/core_stage.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/core_stage.md)
-- [graph/core_graph.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/core_graph.md)
-- [observability/core_progress.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/observability/core_progress.md)
-- [runtime/core_metrics.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/core_metrics.md)
-- [runtime/core_queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/core_queue.md)
-- [stage/core_stages.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/core_stages.md)
-- [observability/core_report.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/observability/core_report.md)
-- [graph/core_structure.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/core_structure.md)
-- [web/core_server.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/web/core_server.md)
-- [other/go_worker.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/other/go_worker.md)
+- [stage/core_executor.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/stage/core_executor.md)
+- [stage/core_stage.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/stage/core_stage.md)
+- [graph/core_graph.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/graph/core_graph.md)
+- [observability/core_progress.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/observability/core_progress.md)
+- [runtime/core_metrics.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/runtime/core_metrics.md)
+- [runtime/core_queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/runtime/core_queue.md)
+- [stage/core_stages.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/stage/core_stages.md)
+- [observability/core_report.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/observability/core_report.md)
+- [graph/core_structure.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/graph/core_structure.md)
+- [web/core_server.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/web/core_server.md)
+- [other/go_worker.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/other/go_worker.md)
 
 推荐阅读顺序:
 
@@ -217,19 +219,31 @@ flowchart TD
 
 以下三篇可以作为补充阅读:
 
-- [runtime/util_queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/util_queue.md)
-- [runtime/util_types.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/util_types.md)
-- [runtime/util_errors.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/util_errors.md)
-- [persistence/core_fail.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/core_fail.md)
-- [persistence/core_log.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/core_log.md)
+- [runtime/util_queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/runtime/util_queue.md)
+- [runtime/util_types.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/runtime/util_types.md)
+- [runtime/util_errors.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/runtime/util_errors.md)
+- [persistence/core_fail.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/persistence/core_fail.md)
+- [persistence/core_log.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/persistence/core_log.md)
 
-如果你更喜欢通过完整案例理解框架的运行方式，可以参考这篇从零开始构建 TaskGraph 的教程：
+如果你更喜欢通过完整案例理解框架的运行方式，可以参考这篇利用 TaskGraph 从零开始构建项目的教程：
 
-[📘案例教程](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/tutorial.md)
+[📘案例教程](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/tutorial.md)
 
 如果你对3.0.7版本加入的ctree_client与其功能感兴趣, 可以看看这一篇:
 
-[📚CelestialTreeClient](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/other/ctree_client.md)
+[📚CelestialTreeClient](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/other/ctree_client.md)
+
+你可以继续运行更多的演示代码，这里记录了各个演示文件与其中的演示函数说明：
+
+[🎮demo/](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/demo/)
+
+​如果你想运行测试代码, 可以先查看如下文档内容:
+
+[🧪tests/](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/tests/)
+
+如果你想查看bench内容, 这里的数据成为框架中部分设计的决策依据:
+
+[⚡bench/](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/bench/)
 
 ## 环境要求（Requirements）
 
@@ -253,24 +267,36 @@ flowchart TD
 <p align="center">
   <img src="https://raw.githubusercontent.com/Mr-xiaotian/CelestialFlow/main/img/file_structure.svg" alt="FileStructure" />
   <br/>
-  <em>celestial-flow 3.1.7</em>
+  <em>celestial-flow 3.1.8</em>
 </p>
 
 (该视图由我的另一个项目[CelestialVault](https://github.com/Mr-xiaotian/CelestialVault)中inst_file.FileTree.print_tree()生成。转换为图片则借助[Carbon](https://carbon.now.sh)。)
 
 ## 版本日志（Version Log）
-- 3.1.7
+- 3.1.8
   - feat:
-    - [Important] 删除executor中的"process"模式, 它跟新的retry机制实在难以匹配;
+    - [Important] 为stage_mode新增"thread"模式, 适合I/O密集型任务和不可pickle的函数;
+      - 在io密集任务的bench测试中, "thread-thread"相比最慢的"serial-serial"快8.7x, 并且优于之前最快的"process-thread";
+    - 新增docs/.../bench与docs/.../demo文档, 补充bench实测结果;
   - refactor:
-    - [Important] 大幅重构retry机制, 现在不再让retry任务重进task_queue, 而是在worker中直接解决;
-      - 模仿CelestialForge中grow的做法;
-    - 大幅重构原本成功与失败结果的缓存机制, 并将原有get_success/error_dict改名为get_success/error_pairs, 以避免task无法作为键时会导致的问题; 
-      - fail数据从fail.jsonl中提取;
-      - success数据从result_queue中直接获取, 为此添加了SuccessSpout作为集合端;
+    - [Important] 删除stage.set_stage_context, 改用graph.connect;
+      - 接口破坏性重构, 需要修改大量graph相关代码;
+    - [Important] name作为第一位参数在stage/executor中强制设定;
+      - 接口破坏性重构
+    - 优化async的executor_mode, 使其同时获取与消费任务, 而非集中消费;
+    - 为上版本retry机制的重构收尾;
   - fix:
-    - 修复部分任务类型无法在log_spout中dump进jsonl文件的问题;
-    - 修复在executor运行时, retry添加任务total数的问题, 虽然这样很帅;
+    - 修复find_unpickleable对bound method的误报;
+    - 修复stage重复检测错误;
+    - 修复retry任务无法被去重的问题;
+  - chore:
+    - 将demo/中所有test_函数名改为demo_函数名;
+    - 重组docs目录结构, 将reference改为src;
+    - docs中新增en/与ja/, 并将原有中文文档移至zh-CN/;
+
+更多过往日志可看:
+
+[change_log.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/runtime/change_log.md )
 
 ## Star 历史趋势（Star History）
 

{celestialflow-3.1.7 → celestialflow-3.1.8}/README.md

@@ -18,6 +18,10 @@
   <img src="https://img.shields.io/badge/Distributed-Worker%20Friendly-orange">
 </p>
 
+<p align="center">
+  <a href="README.md">中文</a> | <a href="docs/en/README.md">English</a> | <a href="docs/ja/README.md">日本語</a>
+</p>
+
 **CelestialFlow** 是一个轻量级但功能完全的任务流框架，适合需要 **复杂依赖关系**、**灵活执行模型**、**跨设备运行**与**实时可视化监控** 的中/大型 Python 任务系统。
 
 - 相比 Airflow/Dagster 更轻、更快开始
@@ -38,9 +42,10 @@ TaskStage 的任务执行模式只有两种：
 * **线性（serial）**
 * **多线程（thread）**
 
-在图级别上，每个 Stage 支持两种上下文模式：
+在图级别上，每个 Stage 支持三种上下文模式：
 
 * **线性执行（serial layout）**：当前节点执行完毕再启动下一节点（下游节点可提前接收任务但不会立即执行）。
+* **线程执行（thread layout）**：当前节点在主进程的独立线程中启动，适合 I/O 密集型任务和不可 pickle 的函数（如 lambda）。
 * **并行执行（process layout）**：当前节点启动后立刻前去启动下一节点。
 
 TaskGraph 能构建完整的 **有向图结构（Directed Graph）**，不仅支持传统的有向无环图（DAG），也能灵活表达 **树形（Tree）**、**环形（loop）** 乃至于 **完全图（Complete Graph）** 形式的任务依赖。
@@ -127,9 +132,12 @@ if __name__ == "__main__":
     stage2 = TaskStage(square, execution_mode="thread")
 
     # 构建任务图结构
-    stage1.set_graph_context([stage2], stage_mode="process", stage_name="Adder")
-    stage2.set_graph_context([], stage_mode="process", stage_name="Squarer")
-    graph = TaskGraph([stage1])
+    stage1 = TaskStage(func=add, execution_mode="thread", unpack_task_args=True, stage_mode="process", name="Adder")
+    stage2 = TaskStage(func=square, execution_mode="thread", stage_mode="process", name="Squarer")
+
+    graph = TaskGraph()
+    graph.set_stages(root_stages=[stage1], stages=[stage2])
+    graph.connect([stage1], [stage2])
 
     # 初始化任务并启动
     graph.start_graph({stage1.get_tag(): [(1, 2), (3, 4), (5, 6)]})
@@ -137,29 +145,23 @@ if __name__ == "__main__":
 
 注意不要在.ipynb中运行。
 
-👉 想查看完整Quick Start，请见[Quick Start](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/quick_start.md)
+👉 想查看完整Quick Start，请见[Quick Start](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/quick_start.md)
 
 ## 深入阅读（Further Reading）
 
-(以下文档完善中)
-
-你可以继续运行更多的测试代码，这里记录了各个测试文件与其中的测试函数说明：
-
-[📄tests/README.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/tests/README.md)
-
 若你想了解框架的整体结构与核心组件，下面的参考文档会对你有帮助：
 
-- [stage/core_executor.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/core_executor.md)
-- [stage/core_stage.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/core_stage.md)
-- [graph/core_graph.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/core_graph.md)
-- [observability/core_progress.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/observability/core_progress.md)
-- [runtime/core_metrics.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/core_metrics.md)
-- [runtime/core_queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/core_queue.md)
-- [stage/core_stages.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/core_stages.md)
-- [observability/core_report.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/observability/core_report.md)
-- [graph/core_structure.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/core_structure.md)
-- [web/core_server.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/web/core_server.md)
-- [other/go_worker.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/other/go_worker.md)
+- [stage/core_executor.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/stage/core_executor.md)
+- [stage/core_stage.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/stage/core_stage.md)
+- [graph/core_graph.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/graph/core_graph.md)
+- [observability/core_progress.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/observability/core_progress.md)
+- [runtime/core_metrics.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/runtime/core_metrics.md)
+- [runtime/core_queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/runtime/core_queue.md)
+- [stage/core_stages.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/stage/core_stages.md)
+- [observability/core_report.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/observability/core_report.md)
+- [graph/core_structure.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/graph/core_structure.md)
+- [web/core_server.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/web/core_server.md)
+- [other/go_worker.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/other/go_worker.md)
 
 推荐阅读顺序:
 
@@ -192,19 +194,31 @@ flowchart TD
 
 以下三篇可以作为补充阅读:
 
-- [runtime/util_queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/util_queue.md)
-- [runtime/util_types.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/util_types.md)
-- [runtime/util_errors.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/util_errors.md)
-- [persistence/core_fail.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/core_fail.md)
-- [persistence/core_log.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/core_log.md)
+- [runtime/util_queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/runtime/util_queue.md)
+- [runtime/util_types.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/runtime/util_types.md)
+- [runtime/util_errors.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/runtime/util_errors.md)
+- [persistence/core_fail.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/persistence/core_fail.md)
+- [persistence/core_log.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/persistence/core_log.md)
 
-如果你更喜欢通过完整案例理解框架的运行方式，可以参考这篇从零开始构建 TaskGraph 的教程：
+如果你更喜欢通过完整案例理解框架的运行方式，可以参考这篇利用 TaskGraph 从零开始构建项目的教程：
 
-[📘案例教程](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/tutorial.md)
+[📘案例教程](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/tutorial.md)
 
 如果你对3.0.7版本加入的ctree_client与其功能感兴趣, 可以看看这一篇:
 
-[📚CelestialTreeClient](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/other/ctree_client.md)
+[📚CelestialTreeClient](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/other/ctree_client.md)
+
+你可以继续运行更多的演示代码，这里记录了各个演示文件与其中的演示函数说明：
+
+[🎮demo/](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/demo/)
+
+​如果你想运行测试代码, 可以先查看如下文档内容:
+
+[🧪tests/](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/tests/)
+
+如果你想查看bench内容, 这里的数据成为框架中部分设计的决策依据:
+
+[⚡bench/](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/bench/)
 
 ## 环境要求（Requirements）
 
@@ -228,24 +242,36 @@ flowchart TD
 <p align="center">
   <img src="https://raw.githubusercontent.com/Mr-xiaotian/CelestialFlow/main/img/file_structure.svg" alt="FileStructure" />
   <br/>
-  <em>celestial-flow 3.1.7</em>
+  <em>celestial-flow 3.1.8</em>
 </p>
 
 (该视图由我的另一个项目[CelestialVault](https://github.com/Mr-xiaotian/CelestialVault)中inst_file.FileTree.print_tree()生成。转换为图片则借助[Carbon](https://carbon.now.sh)。)
 
 ## 版本日志（Version Log）
-- 3.1.7
+- 3.1.8
   - feat:
-    - [Important] 删除executor中的"process"模式, 它跟新的retry机制实在难以匹配;
+    - [Important] 为stage_mode新增"thread"模式, 适合I/O密集型任务和不可pickle的函数;
+      - 在io密集任务的bench测试中, "thread-thread"相比最慢的"serial-serial"快8.7x, 并且优于之前最快的"process-thread";
+    - 新增docs/.../bench与docs/.../demo文档, 补充bench实测结果;
   - refactor:
-    - [Important] 大幅重构retry机制, 现在不再让retry任务重进task_queue, 而是在worker中直接解决;
-      - 模仿CelestialForge中grow的做法;
-    - 大幅重构原本成功与失败结果的缓存机制, 并将原有get_success/error_dict改名为get_success/error_pairs, 以避免task无法作为键时会导致的问题; 
-      - fail数据从fail.jsonl中提取;
-      - success数据从result_queue中直接获取, 为此添加了SuccessSpout作为集合端;
+    - [Important] 删除stage.set_stage_context, 改用graph.connect;
+      - 接口破坏性重构, 需要修改大量graph相关代码;
+    - [Important] name作为第一位参数在stage/executor中强制设定;
+      - 接口破坏性重构
+    - 优化async的executor_mode, 使其同时获取与消费任务, 而非集中消费;
+    - 为上版本retry机制的重构收尾;
   - fix:
-    - 修复部分任务类型无法在log_spout中dump进jsonl文件的问题;
-    - 修复在executor运行时, retry添加任务total数的问题, 虽然这样很帅;
+    - 修复find_unpickleable对bound method的误报;
+    - 修复stage重复检测错误;
+    - 修复retry任务无法被去重的问题;
+  - chore:
+    - 将demo/中所有test_函数名改为demo_函数名;
+    - 重组docs目录结构, 将reference改为src;
+    - docs中新增en/与ja/, 并将原有中文文档移至zh-CN/;
+
+更多过往日志可看:
+
+[change_log.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/zh-CN/src/runtime/change_log.md )
 
 ## Star 历史趋势（Star History）
 

{celestialflow-3.1.7 → celestialflow-3.1.8}/pyproject.toml

@@ -1,81 +1,81 @@
-[build-system]
-requires = ["setuptools>=61.0", "wheel"]
-build-backend = "setuptools.build_meta"
-
-[project]
-name = "celestialflow"                   
-version = "3.1.7"
-description = "A flexible GRAPH-based task orchestration framework."
-readme = "README.md"               
-license = { text = "MIT" }    
-authors = [{ name = "Mr-xiaotian", email = "mingxiaomingtian@gmail.com" }]
-keywords = ["workflow", "task", "graph", "async", "CelestialFlow"]
-requires-python = ">=3.10"
-
-dependencies = [
-    "tqdm",
-    "fastapi",
-    "uvicorn",
-    "requests",
-    "networkx",
-    "redis",
-    "jinja2",
-    "celestialtree>=0.1.2",
-]
-
-classifiers = [
-    "Programming Language :: Python :: 3",
-    "License :: OSI Approved :: MIT License",
-    "Operating System :: OS Independent",
-    "Framework :: FastAPI",
-    "Topic :: Software Development :: Libraries",
-]
-
-[project.urls]
-"Homepage" = "https://github.com/Mr-xiaotian/CelestialFlow"
-"Bug Tracker" = "https://github.com/Mr-xiaotian/CelestialFlow/issues"
-
-[project.scripts]
-celestialflow-web = "celestialflow.web.core_server:main_entry"
-
-[tool.setuptools]
-license-files = []
-
-[tool.setuptools.package-data]
-celestialflow = [
-    "web/templates/**/*.html",
-    "web/static/**/*",
-    "web/config.json"
-]
-
-[tool.pytest.ini_options]
-addopts = "-s"
-log_cli = true
-log_cli_level = "INFO"
-log_cli_format = "%(asctime)s [%(levelname)s] %(message)s"
-log_cli_date_format = "%Y-%m-%d %H:%M:%S"
-
-filterwarnings = [
-    "ignore::DeprecationWarning",
-    "ignore::pytest.PytestDeprecationWarning",
-]
-
-asyncio_default_fixture_loop_scope = "function"
-
-[dependency-groups]
-dev = [
-    "build>=1.2.2.post1",
-    "twine>=6.1.0",
-    "pytest>=8.3.4",
-    "pytest-asyncio>=0.25.3",
-    "httpx>=0.28.1",
-    "python-dotenv>=1.2.2",
-    "dill",
-    "mypy>=1.19.1",
-    "ruff",
-]
-
-[tool.mypy]
-python_version = "3.10"
-no_implicit_optional = false
-disable_error_code = ["import-untyped"]
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "celestialflow"                   
+version = "3.1.8"
+description = "A flexible GRAPH-based task orchestration framework."
+readme = "README.md"               
+license = { text = "MIT" }    
+authors = [{ name = "Mr-xiaotian", email = "mingxiaomingtian@gmail.com" }]
+keywords = ["workflow", "task", "graph", "async", "CelestialFlow"]
+requires-python = ">=3.10"
+
+dependencies = [
+    "tqdm",
+    "fastapi",
+    "uvicorn",
+    "requests",
+    "networkx",
+    "redis",
+    "jinja2",
+    "celestialtree>=0.1.2",
+]
+
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Framework :: FastAPI",
+    "Topic :: Software Development :: Libraries",
+]
+
+[project.urls]
+"Homepage" = "https://github.com/Mr-xiaotian/CelestialFlow"
+"Bug Tracker" = "https://github.com/Mr-xiaotian/CelestialFlow/issues"
+
+[project.scripts]
+celestialflow-web = "celestialflow.web.core_server:main_entry"
+
+[tool.setuptools]
+license-files = []
+
+[tool.setuptools.package-data]
+celestialflow = [
+    "web/templates/**/*.html",
+    "web/static/**/*",
+    "web/config.json"
+]
+
+[tool.pytest.ini_options]
+addopts = "-s"
+log_cli = true
+log_cli_level = "INFO"
+log_cli_format = "%(asctime)s [%(levelname)s] %(message)s"
+log_cli_date_format = "%Y-%m-%d %H:%M:%S"
+
+filterwarnings = [
+    "ignore::DeprecationWarning",
+    "ignore::pytest.PytestDeprecationWarning",
+]
+
+asyncio_default_fixture_loop_scope = "function"
+
+[dependency-groups]
+dev = [
+    "build>=1.2.2.post1",
+    "twine>=6.1.0",
+    "pytest>=8.3.4",
+    "pytest-asyncio>=0.25.3",
+    "httpx>=0.28.1",
+    "python-dotenv>=1.2.2",
+    "dill",
+    "mypy>=1.19.1",
+    "ruff",
+]
+
+[tool.mypy]
+python_version = "3.10"
+no_implicit_optional = false
+disable_error_code = ["import-untyped"]

celestialflow-3.1.8/src/celestialflow/funnel/core_inlet.py

@@ -0,0 +1,22 @@
+# funnel/core_inlet.py
+from __future__ import annotations
+
+from typing import Any
+
+
+class BaseInlet:
+    """数据收集器基类，负责将记录通过队列发送到对应的监听器。"""
+
+    def __init__(self, queue: Any) -> None:
+        """初始化收集器
+
+        :param queue: 记录队列
+        """
+        self.queue: Any = queue
+
+    def _funnel(self, record: Any) -> None:
+        """将记录放入队列
+
+        :param record: 待发送的记录
+        """
+        self.queue.put(record)

{celestialflow-3.1.7 → celestialflow-3.1.8}/src/celestialflow/funnel/core_spout.py

@@ -11,7 +11,10 @@ from ..runtime.util_types import TERMINATION_SIGNAL, TerminationSignal
 
 
 class BaseSpout:
+    """数据监听器基类，在独立后台线程中消费队列记录。"""
+
     def __init__(self) -> None:
+        """初始化监听器及其内部队列和线程引用。"""
         self.queue: Any = MPQueue()
         self._thread: Thread | None = None
 
@@ -25,6 +28,7 @@ class BaseSpout:
         return None
 
     def start(self) -> None:
+        """启动后台监听线程（若未运行）。"""
         self._before_start()
         if self._thread is None or not self._thread.is_alive():
             self._thread = Thread(target=self._spout, daemon=True)
@@ -39,11 +43,15 @@ class BaseSpout:
                 self._handle_record(record)
             except Empty:
                 continue
+            except Exception:  # ← 新增：防止线程崩溃
+                continue  # 或记录到 stderr，至少不丢后续记录
 
     def get_queue(self) -> MPQueue:
+        """获取监听器的输入队列。"""
         return self.queue
 
     def stop(self) -> None:
+        """发送终止信号并等待后台线程结束。"""
         if self._thread is None:
             return