celestialflow 3.1.1__tar.gz → 3.1.2__tar.gz

{celestialflow-3.1.1 → celestialflow-3.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: celestialflow
-Version: 3.1.1
+Version: 3.1.2
 Summary: A flexible GRAPH-based task orchestration framework.
 Author-email: Mr-xiaotian <mingxiaomingtian@gmail.com>
 License: MIT
@@ -12,7 +12,7 @@ Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
 Classifier: Framework :: FastAPI
 Classifier: Topic :: Software Development :: Libraries
-Requires-Python: >=3.8
+Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 Requires-Dist: tqdm
 Requires-Dist: loguru
@@ -176,19 +176,31 @@ if __name__ == "__main__":
 
 若你想了解框架的整体结构与核心组件，下面的参考文档会对你有帮助：
 
-- [🔧TaskExecutor概念](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/task_executor.md)
-- [🔧TaskStage概念](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/task_stage.md)
-- [🌐TaskGraph概念](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/task_graph.md)
-- [📚Go Worker概念](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/go_worker.md)
+- [stage/executor.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/executor.md)
+- [stage/stage.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/stage.md)
+- [graph/graph.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/graph.md)
+- [runtime/progress.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/progress.md)
+- [runtime/metrics.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/metrics.md)
+- [runtime/queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/queue.md)
+- [stage/nodes.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/nodes.md)
+- [observability/report.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/observability/report.md)
+- [graph/structure.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/structure.md)
+- [web/server.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/web/server.md)
+- [other/go_worker.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/other/go_worker.md)
 
 推荐阅读顺序:
 
 ```mermaid
 flowchart TD
-    classDef whiteNode fill:#ffffff,stroke:#000000,color:#000000;
+    classDef core fill:#e6efff,stroke:#3b82f6,color:#1e3a8a;
+    classDef runtime fill:#e9f8ef,stroke:#22c55e,color:#14532d;
+    classDef structure fill:#fff6e6,stroke:#f59e0b,color:#78350f;
+    classDef execution fill:#f3e8ff,stroke:#a855f7,color:#581c87;
+    classDef web fill:#ffeaea,stroke:#ef4444,color:#7f1d1d;
 
     TM[TaskExecutor.md] --> TS[TaskStage.md] --> TG[TaskGraph.md]
     TM --> TP[TaskProgress.md]
+    TM --> TME[TaskMetrics.md]
 
     TG --> TQ[TaskQueue.md]
     TG --> TN[TaskNodes.md]
@@ -198,15 +210,20 @@ flowchart TD
     TR --> TW[TaskWeb.md]
     TN --> GW[Go Worker.md]
 
-    class TM,TG,TP,TQ,TN,TR,TS,TW,GW,TSR whiteNode;
+    class TM,TS,TG core;
+    class TP,TME runtime;
+    class TSR structure;
+    class TQ,TN,GW execution;
+    class TR,TW web;
 ```
 
 以下三篇可以作为补充阅读:
 
-- [TaskTools](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/task_tools.md)
-- [TaskTypes](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/task_types.md)
-- [TaskLogging](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/task_logging.md)
-- [TaskErrors](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/task_errors.md)
+- [runtime/tools.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/tools.md)
+- [runtime/types.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/types.md)
+- [runtime/errors.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/errors.md)
+- [persistence/fail.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/fail.md)
+- [persistence/log.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/log.md)
 
 如果你更喜欢通过完整案例理解框架的运行方式，可以参考这篇从零开始构建 TaskGraph 的教程：
 
@@ -214,7 +231,7 @@ flowchart TD
 
 如果你对3.0.7版本加入的ctree_client与其功能感兴趣, 可以看看这一篇:
 
-[📚CelestialTreeClient](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/ctree_client.md)
+[📚CelestialTreeClient](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/other/ctree_client.md)
 
 ## 环境要求（Requirements）
 
@@ -270,7 +287,7 @@ flowchart TD
   - 在前端中添加error_id的显示, 为之后显示provenance_tree做准备; 
   - 增加大量warning与error, 用于提醒不规范设置; 
   - 优化前后端中错误数据的传输方式, 在大量错误数据时减少内存消耗; 
-  - 优化TaskLogger中log队列的准入机制; 
+  - 优化LogSinker中log队列的准入机制; 
   - 修改部分Bug;
 - 3.1.0:
   - 新增:
@@ -290,6 +307,18 @@ flowchart TD
   - 修复:
     - 修复节点剩余时间在小于1s时显示0的问题(这很影响判断);
     - 在task_graph中使用"staged"模式时会报错的问题;
+- 3.1.2:
+  - feat:
+    - [Important] 模仿已有Logger逻辑, 建立新的组合结构FailListener和FailSinker, 将原有绑定于TaskReporter的fail持久化在线程中实现; 
+    - 借此让TaskExecutor也可以进行fail持久化, 文件为fallback/{date_str}/{executor_errors}({time_str}).jsonl";
+    - 重构整体项目结构, 现在更加清晰;
+    - 补全docs/reference中全部文档, 并令其符合项目重构后结构
+    - 引入uv进行环境管理;
+    - 将executor中技术逻辑分离为runtime/metrics.py;
+    - 整合TaskQueue中重复逻辑;
+  - fix
+    - 修复前端renderNodeList中参数设置错误;
+    - 修复其他微小bug;
 
 ## Star 历史趋势（Star History）
 

{celestialflow-3.1.1 → celestialflow-3.1.2}/README.md

@@ -150,19 +150,31 @@ if __name__ == "__main__":
 
 若你想了解框架的整体结构与核心组件，下面的参考文档会对你有帮助：
 
-- [🔧TaskExecutor概念](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/task_executor.md)
-- [🔧TaskStage概念](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/task_stage.md)
-- [🌐TaskGraph概念](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/task_graph.md)
-- [📚Go Worker概念](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/go_worker.md)
+- [stage/executor.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/executor.md)
+- [stage/stage.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/stage.md)
+- [graph/graph.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/graph.md)
+- [runtime/progress.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/progress.md)
+- [runtime/metrics.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/metrics.md)
+- [runtime/queue.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/queue.md)
+- [stage/nodes.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/stage/nodes.md)
+- [observability/report.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/observability/report.md)
+- [graph/structure.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/graph/structure.md)
+- [web/server.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/web/server.md)
+- [other/go_worker.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/other/go_worker.md)
 
 推荐阅读顺序:
 
 ```mermaid
 flowchart TD
-    classDef whiteNode fill:#ffffff,stroke:#000000,color:#000000;
+    classDef core fill:#e6efff,stroke:#3b82f6,color:#1e3a8a;
+    classDef runtime fill:#e9f8ef,stroke:#22c55e,color:#14532d;
+    classDef structure fill:#fff6e6,stroke:#f59e0b,color:#78350f;
+    classDef execution fill:#f3e8ff,stroke:#a855f7,color:#581c87;
+    classDef web fill:#ffeaea,stroke:#ef4444,color:#7f1d1d;
 
     TM[TaskExecutor.md] --> TS[TaskStage.md] --> TG[TaskGraph.md]
     TM --> TP[TaskProgress.md]
+    TM --> TME[TaskMetrics.md]
 
     TG --> TQ[TaskQueue.md]
     TG --> TN[TaskNodes.md]
@@ -172,15 +184,20 @@ flowchart TD
     TR --> TW[TaskWeb.md]
     TN --> GW[Go Worker.md]
 
-    class TM,TG,TP,TQ,TN,TR,TS,TW,GW,TSR whiteNode;
+    class TM,TS,TG core;
+    class TP,TME runtime;
+    class TSR structure;
+    class TQ,TN,GW execution;
+    class TR,TW web;
 ```
 
 以下三篇可以作为补充阅读:
 
-- [TaskTools](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/task_tools.md)
-- [TaskTypes](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/task_types.md)
-- [TaskLogging](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/task_logging.md)
-- [TaskErrors](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/task_errors.md)
+- [runtime/tools.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/tools.md)
+- [runtime/types.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/types.md)
+- [runtime/errors.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/runtime/errors.md)
+- [persistence/fail.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/fail.md)
+- [persistence/log.md](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/persistence/log.md)
 
 如果你更喜欢通过完整案例理解框架的运行方式，可以参考这篇从零开始构建 TaskGraph 的教程：
 
@@ -188,7 +205,7 @@ flowchart TD
 
 如果你对3.0.7版本加入的ctree_client与其功能感兴趣, 可以看看这一篇:
 
-[📚CelestialTreeClient](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/ctree_client.md)
+[📚CelestialTreeClient](https://github.com/Mr-xiaotian/CelestialFlow/blob/main/docs/reference/other/ctree_client.md)
 
 ## 环境要求（Requirements）
 
@@ -244,7 +261,7 @@ flowchart TD
   - 在前端中添加error_id的显示, 为之后显示provenance_tree做准备; 
   - 增加大量warning与error, 用于提醒不规范设置; 
   - 优化前后端中错误数据的传输方式, 在大量错误数据时减少内存消耗; 
-  - 优化TaskLogger中log队列的准入机制; 
+  - 优化LogSinker中log队列的准入机制; 
   - 修改部分Bug;
 - 3.1.0:
   - 新增:
@@ -264,6 +281,18 @@ flowchart TD
   - 修复:
     - 修复节点剩余时间在小于1s时显示0的问题(这很影响判断);
     - 在task_graph中使用"staged"模式时会报错的问题;
+- 3.1.2:
+  - feat:
+    - [Important] 模仿已有Logger逻辑, 建立新的组合结构FailListener和FailSinker, 将原有绑定于TaskReporter的fail持久化在线程中实现; 
+    - 借此让TaskExecutor也可以进行fail持久化, 文件为fallback/{date_str}/{executor_errors}({time_str}).jsonl";
+    - 重构整体项目结构, 现在更加清晰;
+    - 补全docs/reference中全部文档, 并令其符合项目重构后结构
+    - 引入uv进行环境管理;
+    - 将executor中技术逻辑分离为runtime/metrics.py;
+    - 整合TaskQueue中重复逻辑;
+  - fix
+    - 修复前端renderNodeList中参数设置错误;
+    - 修复其他微小bug;
 
 ## Star 历史趋势（Star History）
 
@@ -277,4 +306,4 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
 ## 作者（Author）
 Author: Mr-xiaotian
 Email: mingxiaomingtian@gmail.com
-Project Link: [https://github.com/Mr-xiaotian/CelestialFlow](https://github.com/Mr-xiaotian/CelestialFlow)
+Project Link: [https://github.com/Mr-xiaotian/CelestialFlow](https://github.com/Mr-xiaotian/CelestialFlow)

{celestialflow-3.1.1 → celestialflow-3.1.2}/pyproject.toml

@@ -4,13 +4,13 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "celestialflow"                   
-version = "3.1.1"                         
+version = "3.1.2"                         
 description = "A flexible GRAPH-based task orchestration framework."
 readme = "README.md"               
 license = { text = "MIT" }    
 authors = [{ name = "Mr-xiaotian", email = "mingxiaomingtian@gmail.com" }]
 keywords = ["workflow", "task", "graph", "async", "CelestialFlow"]
-requires-python = ">=3.8"
+requires-python = ">=3.10"
 
 dependencies = [
     "tqdm",
@@ -37,7 +37,7 @@ classifiers = [
 "Bug Tracker" = "https://github.com/Mr-xiaotian/CelestialFlow/issues"
 
 [project.scripts]
-celestialflow-web = "celestialflow.task_web:main_entry"
+celestialflow-web = "celestialflow.web.server:main_entry"
 
 [tool.setuptools]
 license-files = []
@@ -63,4 +63,9 @@ asyncio_default_fixture_loop_scope = "function"
 dev = [
     "build>=1.2.2.post1",
     "twine>=6.1.0",
+    "pytest>=8.3.4",
+    "pytest-asyncio>=0.25.3",
+    "httpx>=0.28.1",
+    "python-dotenv>=1.2.2",
+    "black>=26.1.0",
 ]

{celestialflow-3.1.1 → celestialflow-3.1.2}/src/celestialflow/__init__.py

@@ -1,14 +1,15 @@
-from .task_graph import TaskGraph
-from .task_executor import TaskExecutor
-from .task_stage import TaskStage
-from .task_nodes import (
+# __init__.py
+from .stage import (
+    TaskExecutor,
+    TaskStage,
     TaskSplitter,
     TaskRedisSink,
     TaskRedisSource,
     TaskRedisAck,
     TaskRouter,
 )
-from .task_structure import (
+from .graph import (
+    TaskGraph,
     TaskChain,
     TaskLoop,
     TaskCross,
@@ -16,15 +17,15 @@ from .task_structure import (
     TaskWheel,
     TaskGrid,
 )
-from .task_types import TerminationSignal
-from .task_tools import (
+from .runtime.types import TerminationSignal
+from .runtime.hash import make_hashable
+from .utils.format import format_table
+from .persistence.jsonl import (
     load_jsonl_logs,
     load_task_by_stage,
     load_task_by_error,
-    make_hashable,
-    format_table,
 )
-from .task_web import TaskWebServer
+from .web.server import TaskWebServer
 
 __all__ = [
     "TaskGraph",

celestialflow-3.1.2/src/celestialflow/graph/__init__.py

@@ -0,0 +1,20 @@
+# graph/__init__.py
+from .graph import TaskGraph
+from .structure import (
+    TaskChain,
+    TaskLoop,
+    TaskCross,
+    TaskComplete,
+    TaskWheel,
+    TaskGrid,
+)
+
+__all__ = [
+    "TaskGraph",
+    "TaskChain",
+    "TaskLoop",
+    "TaskCross",
+    "TaskComplete",
+    "TaskWheel",
+    "TaskGrid",
+]

celestialflow-3.1.2/src/celestialflow/graph/analysis.py

@@ -0,0 +1,50 @@
+# graph/analysis.py
+import networkx as nx
+from networkx import is_directed_acyclic_graph
+from typing import Dict, Any, List
+
+
+# ======== (图论分析) ========
+def format_networkx_graph(structure_graph: List[Dict[str, Any]]) -> nx.DiGraph:
+    """
+    将结构图（由 build_structure_graph 生成）转换为 networkx 有向图（DiGraph）
+
+    :param structure_graph: JSON 格式的任务结构图，List[Dict]
+    :return: 构建好的 networkx.DiGraph
+    """
+    G = nx.DiGraph()
+
+    def add_node_and_edges(node: Dict[str, Any]):
+        node_id = f'{node["stage_name"]}[{node["func_name"]}]'
+        G.add_node(node_id, **{"mode": node.get("stage_mode")})
+
+        for child in node.get("next_stages", []):
+            child_id = f'{child["stage_name"]}[{child["func_name"]}]'
+            G.add_edge(node_id, child_id)
+            # 递归添加子节点
+            add_node_and_edges(child)
+
+    for root in structure_graph:
+        add_node_and_edges(root)
+
+    return G
+
+
+def compute_node_levels(G: nx.DiGraph) -> Dict[str, int]:
+    """
+    计算 DAG 中每个节点的层级（最早执行阶段）
+    前提：图必须是有向无环图（DAG）
+
+    :param G: networkx 有向图（DiGraph）
+    :return: dict[node] = level (int)
+    """
+    if not nx.is_directed_acyclic_graph(G):
+        raise ValueError("该图不是 DAG，无法进行层级划分")
+
+    level = {node: 0 for node in G.nodes}  # 初始层级为 0
+
+    for node in nx.topological_sort(G):  # 按拓扑顺序遍历
+        for succ in G.successors(node):
+            level[succ] = max(level[succ], level[node] + 1)
+
+    return level