celestialflow 3.1.8__tar.gz → 3.1.9__tar.gz

{celestialflow-3.1.8 → celestialflow-3.1.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: celestialflow
-Version: 3.1.8
+Version: 3.1.9
 Summary: A flexible GRAPH-based task orchestration framework.
 Author-email: Mr-xiaotian <mingxiaomingtian@gmail.com>
 License: MIT
@@ -62,10 +62,7 @@ TaskExecutor 实现了对任务的结果缓存，任务去重，进度条显示
 
 但除去直接使用 TaskExecutor，更重要的是使用其子类**TaskStage**。TaskStage 可以互相连接，形成具有上游与下游依赖关系的任务图（**TaskGraph**）。下游 stage 会自动接收上游执行完成的结果作为输入，从而形成明确的数据流。
 
-TaskStage 的任务执行模式只有两种：
-
-* **线性（serial）**
-* **多线程（thread）**
+TaskStage 的任务执行模式同样包含三种，与TaskExecutor中一致。
 
 在图级别上，每个 Stage 支持三种上下文模式：
 
@@ -153,13 +150,10 @@ def square(x):
 
 if __name__ == "__main__":
     # 定义两个任务节点
-    stage1 = TaskStage(add, execution_mode="thread", unpack_task_args=True)
-    stage2 = TaskStage(square, execution_mode="thread")
+    stage1 = TaskStage(name="Adder", func=add, execution_mode="thread", unpack_task_args=True, stage_mode="process")
+    stage2 = TaskStage(name="Squarer"func=square, execution_mode="thread", stage_mode="process")
 
     # 构建任务图结构
-    stage1 = TaskStage(func=add, execution_mode="thread", unpack_task_args=True, stage_mode="process", name="Adder")
-    stage2 = TaskStage(func=square, execution_mode="thread", stage_mode="process", name="Squarer")
-
     graph = TaskGraph()
     graph.set_stages(root_stages=[stage1], stages=[stage2])
     graph.connect([stage1], [stage2])
@@ -253,12 +247,12 @@ flowchart TD
 | 依赖包           | 说明 |
 | ----------------- | ---- |
 | **Python ≥ 3.10**  | 运行环境，建议使用 3.10 及以上版本 |
-| **tqdm**          | 控制台进度条显示，用于任务执行可视化 |
 | **fastapi**       | Web 服务接口框架（用于任务可视化与远程控制） |
 | **uvicorn**       | FastAPI 的高性能 ASGI 服务器 |
 | **requests**      | HTTP 客户端库，用于任务状态上报与远程调用 |
 | **networkx**      | 任务图（TaskGraph）结构与依赖分析 |
 | **jinja2**        | FastAPI 模板引擎，用于 Web 可视化界面渲染 |
+| **tqdm**          | 可选组件，进度条显示，用于任务执行可视化 |
 | **redis**         | 可选组件，用于分布式任务通信（`TaskRedis*` 系列模块） |
 | **celestialtree** | 可选组件，用于任务状态上报与远程调用（`ctree_client`） |
 
@@ -267,32 +261,29 @@ flowchart TD
 <p align="center">
   <img src="https://raw.githubusercontent.com/Mr-xiaotian/CelestialFlow/main/img/file_structure.svg" alt="FileStructure" />
   <br/>
-  <em>celestial-flow 3.1.8</em>
+  <em>celestial-flow 3.1.9</em>
 </p>
 
 (该视图由我的另一个项目[CelestialVault](https://github.com/Mr-xiaotian/CelestialVault)中inst_file.FileTree.print_tree()生成。转换为图片则借助[Carbon](https://carbon.now.sh)。)
 
 ## 版本日志（Version Log）
-- 3.1.8
-  - feat:
-    - [Important] 为stage_mode新增"thread"模式, 适合I/O密集型任务和不可pickle的函数;
-      - 在io密集任务的bench测试中, "thread-thread"相比最慢的"serial-serial"快8.7x, 并且优于之前最快的"process-thread";
-    - 新增docs/.../bench与docs/.../demo文档, 补充bench实测结果;
-  - refactor:
-    - [Important] 删除stage.set_stage_context, 改用graph.connect;
-      - 接口破坏性重构, 需要修改大量graph相关代码;
-    - [Important] name作为第一位参数在stage/executor中强制设定;
-      - 接口破坏性重构
-    - 优化async的executor_mode, 使其同时获取与消费任务, 而非集中消费;
-    - 为上版本retry机制的重构收尾;
-  - fix:
-    - 修复find_unpickleable对bound method的误报;
-    - 修复stage重复检测错误;
-    - 修复retry任务无法被去重的问题;
+- 3.1.9
+ - feat:
+   - [Important] 在TaskStage中允许使用 `async` executor_mode;
+     - 3.1.8版本中其实已经可以实现, 在3.1.9版本中彻底测试通过;
+   - [Important] 创建基类BaseObservor, 现在可以从外界接入自定义Observor来检测运行状况;
+     - 同时创建轻量观察者CallbackObserver;
+   - 在TaskStage中添加init字段`log_level`, 自此stage init字段集实现对executor init字段集的包含关系, 仅多出`stage_mode`字段;
+   - escapeHtml中添加更多替换项;
+   - 在dispatch_thread中添加对futures的清理, 避免无用future的堆积;
+     - 基于bench\bench_futures_memory.py, 在十万级任务下futures内存开销为177.01 MB, 在定期清理后则为0.48 MB
+ - refactor:
+    - 重构TaskProgress, 使其继承BaseObservor;
+      - 接口破坏性重构, TaskExecutor的show_progress参数已删除, 现在只有显性的add_observer TaskProgress后才会使用progress
+      - 具体使用参考demo\demo_executor.py
+    - 重构TaskEnvelope, 不再使用wrap与unwrap, 同时hash值惰性求解;
   - chore:
-    - 将demo/中所有test_函数名改为demo_函数名;
-    - 重组docs目录结构, 将reference改为src;
-    - docs中新增en/与ja/, 并将原有中文文档移至zh-CN/;
+    - 删除部分无用代码以提高代码质量;
 
 更多过往日志可看:
 

{celestialflow-3.1.8 → celestialflow-3.1.9}/README.md

@@ -37,10 +37,7 @@ TaskExecutor 实现了对任务的结果缓存，任务去重，进度条显示
 
 但除去直接使用 TaskExecutor，更重要的是使用其子类**TaskStage**。TaskStage 可以互相连接，形成具有上游与下游依赖关系的任务图（**TaskGraph**）。下游 stage 会自动接收上游执行完成的结果作为输入，从而形成明确的数据流。
 
-TaskStage 的任务执行模式只有两种：
-
-* **线性（serial）**
-* **多线程（thread）**
+TaskStage 的任务执行模式同样包含三种，与TaskExecutor中一致。
 
 在图级别上，每个 Stage 支持三种上下文模式：
 
@@ -128,13 +125,10 @@ def square(x):
 
 if __name__ == "__main__":
     # 定义两个任务节点
-    stage1 = TaskStage(add, execution_mode="thread", unpack_task_args=True)
-    stage2 = TaskStage(square, execution_mode="thread")
+    stage1 = TaskStage(name="Adder", func=add, execution_mode="thread", unpack_task_args=True, stage_mode="process")
+    stage2 = TaskStage(name="Squarer"func=square, execution_mode="thread", stage_mode="process")
 
     # 构建任务图结构
-    stage1 = TaskStage(func=add, execution_mode="thread", unpack_task_args=True, stage_mode="process", name="Adder")
-    stage2 = TaskStage(func=square, execution_mode="thread", stage_mode="process", name="Squarer")
-
     graph = TaskGraph()
     graph.set_stages(root_stages=[stage1], stages=[stage2])
     graph.connect([stage1], [stage2])
@@ -228,12 +222,12 @@ flowchart TD
 | 依赖包           | 说明 |
 | ----------------- | ---- |
 | **Python ≥ 3.10**  | 运行环境，建议使用 3.10 及以上版本 |
-| **tqdm**          | 控制台进度条显示，用于任务执行可视化 |
 | **fastapi**       | Web 服务接口框架（用于任务可视化与远程控制） |
 | **uvicorn**       | FastAPI 的高性能 ASGI 服务器 |
 | **requests**      | HTTP 客户端库，用于任务状态上报与远程调用 |
 | **networkx**      | 任务图（TaskGraph）结构与依赖分析 |
 | **jinja2**        | FastAPI 模板引擎，用于 Web 可视化界面渲染 |
+| **tqdm**          | 可选组件，进度条显示，用于任务执行可视化 |
 | **redis**         | 可选组件，用于分布式任务通信（`TaskRedis*` 系列模块） |
 | **celestialtree** | 可选组件，用于任务状态上报与远程调用（`ctree_client`） |
 
@@ -242,32 +236,29 @@ flowchart TD
 <p align="center">
   <img src="https://raw.githubusercontent.com/Mr-xiaotian/CelestialFlow/main/img/file_structure.svg" alt="FileStructure" />
   <br/>
-  <em>celestial-flow 3.1.8</em>
+  <em>celestial-flow 3.1.9</em>
 </p>
 
 (该视图由我的另一个项目[CelestialVault](https://github.com/Mr-xiaotian/CelestialVault)中inst_file.FileTree.print_tree()生成。转换为图片则借助[Carbon](https://carbon.now.sh)。)
 
 ## 版本日志（Version Log）
-- 3.1.8
-  - feat:
-    - [Important] 为stage_mode新增"thread"模式, 适合I/O密集型任务和不可pickle的函数;
-      - 在io密集任务的bench测试中, "thread-thread"相比最慢的"serial-serial"快8.7x, 并且优于之前最快的"process-thread";
-    - 新增docs/.../bench与docs/.../demo文档, 补充bench实测结果;
-  - refactor:
-    - [Important] 删除stage.set_stage_context, 改用graph.connect;
-      - 接口破坏性重构, 需要修改大量graph相关代码;
-    - [Important] name作为第一位参数在stage/executor中强制设定;
-      - 接口破坏性重构
-    - 优化async的executor_mode, 使其同时获取与消费任务, 而非集中消费;
-    - 为上版本retry机制的重构收尾;
-  - fix:
-    - 修复find_unpickleable对bound method的误报;
-    - 修复stage重复检测错误;
-    - 修复retry任务无法被去重的问题;
+- 3.1.9
+ - feat:
+   - [Important] 在TaskStage中允许使用 `async` executor_mode;
+     - 3.1.8版本中其实已经可以实现, 在3.1.9版本中彻底测试通过;
+   - [Important] 创建基类BaseObservor, 现在可以从外界接入自定义Observor来检测运行状况;
+     - 同时创建轻量观察者CallbackObserver;
+   - 在TaskStage中添加init字段`log_level`, 自此stage init字段集实现对executor init字段集的包含关系, 仅多出`stage_mode`字段;
+   - escapeHtml中添加更多替换项;
+   - 在dispatch_thread中添加对futures的清理, 避免无用future的堆积;
+     - 基于bench\bench_futures_memory.py, 在十万级任务下futures内存开销为177.01 MB, 在定期清理后则为0.48 MB
+ - refactor:
+    - 重构TaskProgress, 使其继承BaseObservor;
+      - 接口破坏性重构, TaskExecutor的show_progress参数已删除, 现在只有显性的add_observer TaskProgress后才会使用progress
+      - 具体使用参考demo\demo_executor.py
+    - 重构TaskEnvelope, 不再使用wrap与unwrap, 同时hash值惰性求解;
   - chore:
-    - 将demo/中所有test_函数名改为demo_函数名;
-    - 重组docs目录结构, 将reference改为src;
-    - docs中新增en/与ja/, 并将原有中文文档移至zh-CN/;
+    - 删除部分无用代码以提高代码质量;
 
 更多过往日志可看:
 

{celestialflow-3.1.8 → celestialflow-3.1.9}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "celestialflow"                   
-version = "3.1.8"
+version = "3.1.9"
 description = "A flexible GRAPH-based task orchestration framework."
 readme = "README.md"               
 license = { text = "MIT" }    

{celestialflow-3.1.8 → celestialflow-3.1.9}/src/celestialflow/__init__.py

@@ -8,6 +8,7 @@ from .graph import (
     TaskLoop,
     TaskWheel,
 )
+from .observability import BaseObserver, CallbackObserver, TaskProgress
 from .persistence.util_jsonl import (
     load_jsonl_logs,
     load_task_by_error,
@@ -26,7 +27,7 @@ from .stage import (
 )
 from .utils.util_benchmark import benchmark_executor, benchmark_graph
 from .utils.util_format import format_table
-from .web.core_server import TaskWebServer
+from .web import TaskWebServer
 
 __all__ = [
     "TaskGraph",
@@ -36,6 +37,9 @@ __all__ = [
     "TaskComplete",
     "TaskWheel",
     "TaskGrid",
+    "BaseObserver",
+    "CallbackObserver",
+    "TaskProgress",
     "TaskExecutor",
     "TaskStage",
     "TaskSplitter",

{celestialflow-3.1.8 → celestialflow-3.1.9}/src/celestialflow/graph/core_graph.py

@@ -423,7 +423,7 @@ class TaskGraph:
                     CTreeEvent.TASK_INPUT,
                     payload=stage.get_summary(),
                 )
-                envelope = TaskEnvelope.wrap(task, input_id, source="input")
+                envelope = TaskEnvelope(task, input_id, source="input")
                 in_queue.put(envelope)
                 input_ids.add(input_id)
 

{celestialflow-3.1.8 → celestialflow-3.1.9}/src/celestialflow/observability/__init__.py

@@ -1,10 +1,12 @@
 # observability/__init__.py
-from .core_progress import NullTaskProgress, TaskProgress
+from .core_observer import BaseObserver, CallbackObserver
+from .core_progress import TaskProgress
 from .core_report import NullTaskReporter, TaskReporter
 
 __all__ = [
+    "BaseObserver",
+    "CallbackObserver",
     "TaskReporter",
     "NullTaskReporter",
     "TaskProgress",
-    "NullTaskProgress",
 ]

celestialflow-3.1.9/src/celestialflow/observability/core_observer.py

@@ -0,0 +1,26 @@
+# observability/core_observer.py
+from __future__ import annotations
+
+
+class BaseObserver:
+    """执行器生命周期观察者基类，子类按需覆写。"""
+
+    def on_start(self, name: str, total: int) -> None: ...
+
+    def on_task_success(self, count: int = 1) -> None: ...
+
+    def on_task_fail(self, count: int = 1) -> None: ...
+
+    def on_task_duplicate(self, count: int = 1) -> None: ...
+
+    def on_tasks_added(self, count: int) -> None: ...
+
+    def on_finish(self) -> None: ...
+
+
+class CallbackObserver(BaseObserver):
+    """通过回调函数创建的轻量观察者，无需定义子类。"""
+
+    def __init__(self, **callbacks):
+        for name, fn in callbacks.items():
+            setattr(self, name, fn)

celestialflow-3.1.9/src/celestialflow/observability/core_progress.py

@@ -0,0 +1,30 @@
+# observability/core_progress.py
+from __future__ import annotations
+
+from tqdm import tqdm
+
+from .core_observer import BaseObserver
+
+
+class TaskProgress(BaseObserver):
+    """基于 tqdm 的进度条观察者"""
+
+    def on_start(self, name: str, total: int) -> None:
+        self._bar = tqdm(total=total, desc=name)
+
+    def on_task_success(self, count: int = 1) -> None:
+        self._bar.update(count)
+
+    def on_task_fail(self, count: int = 1) -> None:
+        self._bar.update(count)
+
+    def on_task_duplicate(self, count: int = 1) -> None:
+        self._bar.update(count)
+
+    def on_tasks_added(self, count: int) -> None:
+        if count:
+            self._bar.total += count
+            self._bar.refresh()
+
+    def on_finish(self) -> None:
+        self._bar.close()

{celestialflow-3.1.8 → celestialflow-3.1.9}/src/celestialflow/observability/core_report.py

@@ -109,9 +109,11 @@ class TaskReporter:
             res = self._session.get(
                 f"{self.base_url}/api/pull_interval", timeout=self._pull_timeout()
             )
-            if res.ok:
-                interval = res.json().get("interval", 5)
-                self.interval = max(1.0, min(interval, 60.0))
+            if not res.ok:
+                raise RuntimeError(f"Failed to pull interval: {res.status_code}")
+            
+            interval = res.json().get("interval", 5)
+            self.interval = max(1.0, min(interval, 60.0))
         except Exception as e:
             self.log_inlet.pull_interval_failed(e)
 
@@ -121,9 +123,11 @@ class TaskReporter:
             res = self._session.get(
                 f"{self.base_url}/api/pull_history_limit", timeout=self._pull_timeout()
             )
-            if res.ok:
-                history_limit = res.json().get("historyLimit", 20)
-                self.history_limit = max(1, min(history_limit, 100))
+            if not res.ok:
+                raise RuntimeError(f"Failed to pull history limit: {res.status_code}")
+            
+            history_limit = res.json().get("historyLimit", 20)
+            self.history_limit = max(1, min(history_limit, 100))
         except Exception as e:
             self.log_inlet.pull_history_limit_failed(e)
 
@@ -133,26 +137,28 @@ class TaskReporter:
             res = self._session.get(
                 f"{self.base_url}/api/pull_task_injection", timeout=self._pull_timeout()
             )
-            if res.ok:
-                injection_tasks: list[dict] = res.json()
-                for injection in injection_tasks:
-                    target_stage = injection.get("node")
-                    task_datas = injection.get("task_datas")
-                    if target_stage is None or task_datas is None:
-                        continue
-
-                    # 这里你可以按需注入到不同的节点
-                    task_datas = [
-                        task if task != "TERMINATION_SIGNAL" else TERMINATION_SIGNAL
-                        for task in task_datas
-                    ]
-                    try:
-                        self.task_graph.put_stage_queue(
-                            {target_stage: task_datas}, put_termination_signal=False
-                        )
-                        self.log_inlet.inject_tasks_success(target_stage, task_datas)
-                    except Exception as e:
-                        self.log_inlet.inject_tasks_failed(target_stage, task_datas, e)
+            if not res.ok:
+                raise RuntimeError(f"Failed to pull task injection: {res.status_code}")
+            
+            injection_tasks: list[dict] = res.json()
+            for injection in injection_tasks:
+                target_stage = injection.get("node")
+                task_datas = injection.get("task_datas")
+                if target_stage is None or task_datas is None:
+                    continue
+
+                # 这里你可以按需注入到不同的节点
+                task_datas = [
+                    task if task != "TERMINATION_SIGNAL" else TERMINATION_SIGNAL
+                    for task in task_datas
+                ]
+                try:
+                    self.task_graph.put_stage_queue(
+                        {target_stage: task_datas}, put_termination_signal=False
+                    )
+                    self.log_inlet.inject_tasks_success(target_stage, task_datas)
+                except Exception as e:
+                    self.log_inlet.inject_tasks_failed(target_stage, task_datas, e)
         except Exception as e:
             self.log_inlet.pull_tasks_failed(e)
 
@@ -162,6 +168,7 @@ class TaskReporter:
         try:
             current_rev = self.task_graph.get_total_error_num()
             jsonl_path = self.task_graph.get_fallback_path()
+            resp = {"ok": True, "cached": False}  # 默认响应，避免未定义变量
 
             # 无新增错误，跳过
             if current_rev == self._last_pushed_errors_rev:

{celestialflow-3.1.8 → celestialflow-3.1.9}/src/celestialflow/persistence/core_log.py

@@ -312,17 +312,17 @@ class LogInlet(BaseInlet):
 
     # ==== queue ====
     def put_item(
-        self, item_type: str, item_id: int, left_tag: str, right_tag: str
+        self, item_type: str, item_id: int, in_tag: str, out_tag: str
     ) -> None:
         """记录队列 put 操作"""
-        edge = f"'{left_tag}' -> '{right_tag}'"
+        edge = f"'{in_tag}' -> '{out_tag}'"
         self._log("TRACE", f"Put {item_type}#{item_id} into Edge({edge}).")
 
     def put_item_error(
-        self, left_tag: str, right_tag: str, exception: Exception
+        self, in_tag: str, out_tag: str, exception: Exception
     ) -> None:
         """记录队列 put 失败"""
-        edge = f"'{left_tag}' -> '{right_tag}'"
+        edge = f"'{in_tag}' -> '{out_tag}'"
         exception_text = str(exception).replace("\n", " ")
         self._log(
             "WARNING",
@@ -330,17 +330,17 @@ class LogInlet(BaseInlet):
         )
 
     def get_item(
-        self, item_type: str, item_id: int, left_tag: str, right_tag: str
+        self, item_type: str, item_id: int, in_tag: str, out_tag: str
     ) -> None:
         """记录队列 get 操作"""
-        edge = f"'{left_tag}' -> '{right_tag}'"
+        edge = f"'{in_tag}' -> '{out_tag}'"
         self._log("TRACE", f"Get {item_type}#{item_id} from Edge({edge}).")
 
     def get_item_error(
-        self, left_tag: str, right_tag: str, exception: Exception
+        self, in_tag: str, out_tag: str, exception: Exception
     ) -> None:
         """记录队列 get 失败"""
-        edge = f"'{left_tag}' -> '{right_tag}'"
+        edge = f"'{in_tag}' -> '{out_tag}'"
         exception_text = str(exception).replace("\n", " ")
         self._log(
             "WARNING",
@@ -415,11 +415,11 @@ class LogInlet(BaseInlet):
             f"[Reporter] Push 'structure' failed: {type(exception).__name__}({exception}).",
         )
 
-    def push_topology_failed(self, exception: Exception) -> None:
-        """记录推送拓扑信息失败"""
+    def push_analysis_failed(self, exception: Exception) -> None:
+        """记录推送分析信息失败"""
         self._log(
             "WARNING",
-            f"[Reporter] Push 'topology' failed: {type(exception).__name__}({exception}).",
+            f"[Reporter] Push 'analysis' failed: {type(exception).__name__}({exception}).",
         )
 
     def push_summary_failed(self, exception: Exception) -> None:
@@ -429,13 +429,6 @@ class LogInlet(BaseInlet):
             f"[Reporter] Push 'summary' failed: {type(exception).__name__}({exception}).",
         )
 
-    def push_analysis_failed(self, exception: Exception) -> None:
-        """记录推送分析信息失败"""
-        self._log(
-            "WARNING",
-            f"[Reporter] Push 'analysis' failed: {type(exception).__name__}({exception}).",
-        )
-
     def push_history_failed(self, exception: Exception) -> None:
         """记录推送历史信息失败"""
         self._log(

{celestialflow-3.1.8 → celestialflow-3.1.9}/src/celestialflow/runtime/core_dispatch.py

@@ -73,7 +73,7 @@ class TaskDispatch:
         :param task_envelope: 包含任务信息的信封
         :return: 是否命中重复任务
         """
-        _, task_hash, _ = task_envelope.unwrap()
+        task_hash = task_envelope.get_hash()
 
         if self.task_executor.metrics.is_duplicate(task_hash):
             self.task_executor.deal_duplicate(task_envelope)
@@ -87,7 +87,7 @@ class TaskDispatch:
 
         :param task_envelope: 包含任务信息的信封
         """
-        task, _, _ = task_envelope.unwrap()
+        task = task_envelope.get_task()
         max_retries = self.task_executor.max_retries
 
         for retry_time in range(max_retries + 1):
@@ -114,7 +114,7 @@ class TaskDispatch:
 
         :param task_envelope: 包含任务信息的信封
         """
-        task, _, _ = task_envelope.unwrap()
+        task = task_envelope.get_task()
         max_retries = self.task_executor.max_retries
 
         for retry_time in range(max_retries + 1):
@@ -177,7 +177,10 @@ class TaskDispatch:
 
             if self._pool is None:
                 raise RuntimeError("execution pool has not been initialized")
+            
             futures.append(self._pool.submit(self._worker, envelope))
+            if len(futures) >= self.max_workers * 2:
+                futures = [f for f in futures if not f.done()]
 
         # 等待当前批次的所有任务完成
         for future in futures:

celestialflow-3.1.9/src/celestialflow/runtime/core_envelope.py

@@ -0,0 +1,58 @@
+# runtime/core_envelope.py
+from typing import Any
+
+from .util_hash import object_to_str_hash
+
+
+class TaskEnvelope:
+    """任务信封，封装原始任务及其哈希、ID、来源等元信息。"""
+
+    __slots__ = ("task", "hash", "id", "source", "prev")
+
+    def __init__(self, task: Any, id: int, source: str, prev: Any = None):
+        """
+        :param task: 原始任务
+        :param id: 任务 ID
+        :param source: 任务来源标识
+        :param prev: 前一个任务（用于结果缓存时回溯）
+        """
+        self.task = task
+        self.hash = None
+        self.id = id
+
+        self.source = source
+        self.prev = prev
+    
+    def get_task(self) -> Any:
+        """
+        获取原始任务
+
+        :return: 原始任务
+        """
+        return self.task
+    
+    def get_hash(self) -> str:
+        """
+        获取任务哈希
+
+        :return: 任务哈希
+        """
+        if self.hash is None:
+            self.hash = object_to_str_hash(self.task)
+        return self.hash
+    
+    def get_id(self) -> int:
+        """
+        获取任务 ID
+
+        :return: 任务 ID
+        """
+        return self.id
+
+    def change_id(self, new_id: int) -> None:
+        """
+        修改 id
+
+        :param new_id: 新的任务 id
+        """
+        self.id = new_id

{celestialflow-3.1.8 → celestialflow-3.1.9}/src/celestialflow/runtime/core_metrics.py

@@ -1,5 +1,4 @@
 # runtime/core_metrics.py
-import asyncio
 from threading import Lock
 
 from .util_factories import (
@@ -85,7 +84,7 @@ class TaskMetrics:
     # ==== 去重 ====
     def is_duplicate(self, task_hash: str) -> bool:
         """
-        检查任务是否重复
+        检查任务是否重复, 是原子操作
 
         :param task_hash: 任务的哈希值
         :return: 如果启用了去重检查且任务哈希存在于已处理集合中，返回 True；否则返回 False。
@@ -147,16 +146,6 @@ class TaskMetrics:
         with self.success_counter.get_lock():
             self.success_counter.value += count
 
-    async def add_success_count_async(self, count: int = 1):
-        """
-        异步更新成功任务计数器
-
-        在独立线程中执行 add_success_count，避免阻塞事件循环。
-
-        :param count: 增加的成功任务数量，默认值为 1。
-        """
-        await asyncio.to_thread(self.add_success_count, count)
-
     def add_error_count(self, count: int = 1):
         """
         更新失败任务计数器

{celestialflow-3.1.8 → celestialflow-3.1.9}/src/celestialflow/runtime/core_queue.py

@@ -136,10 +136,15 @@ class TaskInQueue:
         :return: 出队的任务或终止符号id池
         """
         while True:
-            item: TaskEnvelope | TerminationSignal = self.queue.get()
-            result = self._deal_get_item(item)
-            if result is not None:
+            try:
+                item: TaskEnvelope | TerminationSignal = self.queue.get()
+                result = self._deal_get_item(item)
+                if result is None:
+                    continue
                 return result
+            except Exception as e:
+                self.log_inlet.get_item_error("unknown", self.out_tag, e)
+                continue
 
     def _deal_get_item(
         self, item: TaskEnvelope | TerminationSignal

{celestialflow-3.1.8 → celestialflow-3.1.9}/src/celestialflow/runtime/util_factories.py

@@ -57,7 +57,7 @@ def make_queue_backend(mode: str) -> type:
 
 def make_task_in_queue(
     *,
-    queue: any,
+    queue: Any,
     executor: "TaskExecutor",
 ) -> TaskInQueue:
     """
@@ -77,7 +77,7 @@ def make_task_in_queue(
 
 def make_task_out_queue(
     *,
-    queue: any,
+    queue: Any,
     executor: "TaskExecutor",
 ) -> TaskOutQueue:
     """