fluty 0.1.1__tar.gz

fluty-0.1.1/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2024-2026 Jingyuan Tian. All rights reserved.
+
+Except for the excluded lab-owned directories listed below, this repository
+is an individually authored work by Jingyuan Tian and is proprietary and
+confidential.
+
+This notice does not apply to lab-owned projects copied into this repository,
+including:
+- `applications/SAGE`
+- `applications/sage-studio`
+- `applications/sageData`
+
+No permission is granted to use, copy, modify, merge, publish, distribute,
+sublicense, or sell this Software, in whole or in part, in source or binary
+form, except with prior written permission from the copyright holder.
+
+Unauthorized use, disclosure, reproduction, or distribution is strictly
+prohibited.
+
+For licensing requests, contact: jingyuantian@hust.edu.cn

fluty-0.1.1/PKG-INFO

@@ -0,0 +1,91 @@
+Metadata-Version: 2.4
+Name: fluty
+Version: 0.1.1
+Summary: Flownet: A distributed dataflow orchestration system.
+Author-email: Flecther Tian <jingyuan_tian@hust.edu.cn>, SAGE Team <shuhao_zhang@hust.edu.cn>
+License: Proprietary - All Rights Reserved
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: cloudpickle>=2.2
+Requires-Dist: aiohttp
+Requires-Dist: pyyaml
+Requires-Dist: grpcio
+Requires-Dist: rich
+Requires-Dist: fastapi
+Requires-Dist: uvicorn[standard]
+Requires-Dist: pydantic
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: pre-commit; extra == "dev"
+Dynamic: license-file
+
+
+---
+
+# SAGE-Flownet
+
+**SAGE-Flownet** 是一个轻量、原生 Python 的动态数据流执行引擎。它以“嵌套动态图”为核心模型，把用户写下的普通 Python 函数编译为可执行的 FlowTask DAG，并由运行时按需调度执行，让复杂的工作流像调用函数一样自然。
+
+它不是另一个固定模板的流水线框架。Flownet 将远程调用、闭包、回调以及子图嵌套视为**一等公民**，在不牺牲 Python 语义的前提下，支持灵活组合和横向扩展。
+
+## 安装与使用
+
+```bash
+pip install isage-flownet
+```
+
+> **⚠️ PyPI 包名 vs. import 名**
+>
+> SAGE 生态约定：PyPI 包名使用 `isage-` 前缀（因为 `sage` 在 PyPI 已被占用），
+> 但 Python **import 名**使用 `sage.*` 命名空间。
+>
+> | 操作 | 命令 |
+> |------|------|
+> | 安装 | `pip install isage-flownet` |
+> | import | `from flownet import flow, init` |
+>
+> `import isage_flownet` **不存在**，请始终使用 `flownet`。
+
+```python
+from flownet import flow, init, create_actor
+
+@flow
+def my_pipeline(init_stream):
+    return init_stream.map(lambda x: x * 2)
+
+init()  # 启动运行时
+```
+
+## ✧ 项目如何工作（概述）
+
+* **Flow 定义与编译**：通过 `@flow` 装饰器将函数包装为 Flow 定义（`FlowDef`），编译为包含 Transformation 的 FlowTask 图。
+* **数据流建模**：`DataStream.map/flatmap` 等算子创建上下游关系，形成可追踪的依赖图和执行边。
+* **运行时执行**：`FlownetRuntime` 负责处理请求栈（TransformationFrame），并由 operator handler 决定目标调用与转发策略。
+* **Actor 调用统一化**：`ActorRef` / `ActorMethodRef` 将本地与远程调用统一抽象，运行时根据目标地址完成派发。
+* **FlowRun 输出通道**：运行时通过 `FlowRunRef` 的输出通道暴露“写入输入—等待输出”的调用方式，负责 fork/join 逻辑与结果收集。
+
+## ✧ 预期目标与效果
+
+* **保持原生 Python 开发体验**：无需额外 DSL，工作流像写函数一样简单。
+* **支持动态与嵌套的复杂流程**：运行时编译、依赖跟踪与延迟执行适配不确定结构。
+* **可扩展的分布式执行**：为本地/远程统一执行提供清晰路径，便于水平扩展。
+* **更可靠的并发与结果聚合**：由运行时完成 fork/join、收集与异常处理，降低业务代码复杂度。
+
+
+## Intellectual Property
+
+`All Rights Reserved` applies to repository contents authored by Jingyuan Tian,
+including code, documentation, benchmarks, and related assets.
+
+The following copied lab projects are excluded from this claim and retain their
+original ownership/license context: `applications/SAGE`, `applications/sage-studio`,
+`applications/sageData`.
+
+1. License terms: `LICENSE`
+2. Ownership details: `COPYRIGHT.md`
+
+
+---

fluty-0.1.1/README.md

@@ -0,0 +1,67 @@
+
+---
+
+# SAGE-Flownet
+
+**SAGE-Flownet** 是一个轻量、原生 Python 的动态数据流执行引擎。它以“嵌套动态图”为核心模型，把用户写下的普通 Python 函数编译为可执行的 FlowTask DAG，并由运行时按需调度执行，让复杂的工作流像调用函数一样自然。
+
+它不是另一个固定模板的流水线框架。Flownet 将远程调用、闭包、回调以及子图嵌套视为**一等公民**，在不牺牲 Python 语义的前提下，支持灵活组合和横向扩展。
+
+## 安装与使用
+
+```bash
+pip install isage-flownet
+```
+
+> **⚠️ PyPI 包名 vs. import 名**
+>
+> SAGE 生态约定：PyPI 包名使用 `isage-` 前缀（因为 `sage` 在 PyPI 已被占用），
+> 但 Python **import 名**使用 `sage.*` 命名空间。
+>
+> | 操作 | 命令 |
+> |------|------|
+> | 安装 | `pip install isage-flownet` |
+> | import | `from flownet import flow, init` |
+>
+> `import isage_flownet` **不存在**，请始终使用 `flownet`。
+
+```python
+from flownet import flow, init, create_actor
+
+@flow
+def my_pipeline(init_stream):
+    return init_stream.map(lambda x: x * 2)
+
+init()  # 启动运行时
+```
+
+## ✧ 项目如何工作（概述）
+
+* **Flow 定义与编译**：通过 `@flow` 装饰器将函数包装为 Flow 定义（`FlowDef`），编译为包含 Transformation 的 FlowTask 图。
+* **数据流建模**：`DataStream.map/flatmap` 等算子创建上下游关系，形成可追踪的依赖图和执行边。
+* **运行时执行**：`FlownetRuntime` 负责处理请求栈（TransformationFrame），并由 operator handler 决定目标调用与转发策略。
+* **Actor 调用统一化**：`ActorRef` / `ActorMethodRef` 将本地与远程调用统一抽象，运行时根据目标地址完成派发。
+* **FlowRun 输出通道**：运行时通过 `FlowRunRef` 的输出通道暴露“写入输入—等待输出”的调用方式，负责 fork/join 逻辑与结果收集。
+
+## ✧ 预期目标与效果
+
+* **保持原生 Python 开发体验**：无需额外 DSL，工作流像写函数一样简单。
+* **支持动态与嵌套的复杂流程**：运行时编译、依赖跟踪与延迟执行适配不确定结构。
+* **可扩展的分布式执行**：为本地/远程统一执行提供清晰路径，便于水平扩展。
+* **更可靠的并发与结果聚合**：由运行时完成 fork/join、收集与异常处理，降低业务代码复杂度。
+
+
+## Intellectual Property
+
+`All Rights Reserved` applies to repository contents authored by Jingyuan Tian,
+including code, documentation, benchmarks, and related assets.
+
+The following copied lab projects are excluded from this claim and retain their
+original ownership/license context: `applications/SAGE`, `applications/sage-studio`,
+`applications/sageData`.
+
+1. License terms: `LICENSE`
+2. Ownership details: `COPYRIGHT.md`
+
+
+---

fluty-0.1.1/pyproject.toml

@@ -0,0 +1,43 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "fluty"
+version = "0.1.1"
+description = "Flownet: A distributed dataflow orchestration system."
+authors = [{ name = "Flecther Tian", email = "jingyuan_tian@hust.edu.cn" }, { name = "SAGE Team", email = "shuhao_zhang@hust.edu.cn" }]
+license = { text = "Proprietary - All Rights Reserved" }
+
+readme = "README.md"
+requires-python = ">=3.8"
+dependencies = [
+  "cloudpickle>=2.2",
+  "aiohttp",
+  "pyyaml",
+  "grpcio",
+  "rich",
+  # 还有你分布式 actor 或 pipeline 用到的依赖
+  "fastapi",
+  "uvicorn[standard]",
+  "pydantic"
+]
+
+[project.optional-dependencies]
+dev = [
+  "pytest",
+  "black",
+  "mypy",
+  "pre-commit"
+]
+
+[project.scripts]
+flownet = "flownet.cli:main"
+
+# Package discovery (PEP 420 namespace)
+[tool.setuptools.packages.find]
+where = ["src"]
+namespaces = true
+
+
+[project.urls]

fluty-0.1.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

fluty-0.1.1/src/flownet/__init__.py

@@ -0,0 +1,56 @@
+from flownet.api import (
+    flow,
+    init,
+    init_user,
+    create_actor,
+    create_local_actor,
+    flow_program_ref,
+    register_flow_program,
+    list_flow_programs,
+    get_flow_program,
+    delete_flow_program,
+    submit_flow_program,
+    stateless,
+    stateless_op,
+    exception_handler,
+    flow_exception_handler,
+    Dataset,
+    DatasetExecutionMetrics,
+    DatasetWriteResult,
+    SINK_IDEMPOTENCY_META_FIELD,
+    read_rows,
+    read_parquet,
+    read_images,
+    read_tfrecords,
+)
+from flownet.runtime.runtime import  configure_runtime
+__all__ = [
+    "flow",
+    "init",
+    "init_user",
+    "create_actor",
+    "create_local_actor",
+    "flow_program_ref",
+    "register_flow_program",
+    "list_flow_programs",
+    "get_flow_program",
+    "delete_flow_program",
+    "submit_flow_program",
+    "stateless",
+    "stateless_op",
+    "exception_handler",
+    "flow_exception_handler",
+    "configure_runtime",
+    "Dataset",
+    "DatasetExecutionMetrics",
+    "DatasetWriteResult",
+    "SINK_IDEMPOTENCY_META_FIELD",
+    "read_rows",
+    "read_parquet",
+    "read_images",
+    "read_tfrecords",
+]
+
+
+
+# 名字待定，叫做flownet，或者soda(stack-objective distributed architecture)

fluty-0.1.1/src/flownet/api/__init__.py

@@ -0,0 +1,226 @@
+# DOCS_REF:
+# - docs/user/03-programming-model.md
+# - docs/user/reference/api-surface.md
+# - docs/user/cookbook/flow-pipeline-patterns.md
+# DOCS_SYNC:
+# - If exported API names or flow decorator behavior changes, update the
+#   referenced docs and examples/docs/ snippets that import from flownet.api.
+
+from dataclasses import replace
+
+from flownet.api.flow_exception_handlers import (
+    flow_exception_handler,
+    exception_handler,
+)
+from flownet.core.lockfree import lockfree
+from flownet.api.topic_stream import topic_stream, TopicStream, from_topic, from_service
+from flownet.api.stateless_op import stateless, stateless_op, StatelessOpRef, StatelessOpConfig, StatelessScope
+from flownet.api.services import (
+    get_services,
+    try_get_services,
+    start_service,
+    publish_service,
+    stop_service,
+    list_services,
+    start_source,
+    publish_topic,
+    stop_source,
+    list_sources,
+)
+from flownet.api import services as services
+from flownet.api import tasks as tasks
+from flownet.api import node as node
+from flownet.api import node_control as node_control
+from flownet.api import scheduler as scheduler
+
+from flownet.api.runtime import (
+    create_actor,
+    create_local_actor,
+    list_runtime_topics,
+    list_runtime_actors,
+    list_runtime_nodes,
+    flow_program_ref,
+    register_flow_program,
+    list_flow_programs,
+    get_flow_program,
+    delete_flow_program,
+    submit_flow_program,
+)
+from flownet.api import runtime as runtime
+from flownet.runtime.runtime import configure_runtime
+from flownet.config import FlownetConfig, RuntimeConfigInput, coerce_flownet_config
+
+
+# TODO: 里边加一个decorator，然后把func参数放到decorator里边去
+# flow decorator
+def flow(func):
+    from flownet.compiler.flow_def import FlowDef
+    return FlowDef(func, static = True)
+
+
+from flownet.api.dataset import (
+    Dataset,
+    DatasetExecutionMetrics,
+    DatasetWriteResult,
+    SINK_IDEMPOTENCY_META_FIELD,
+    read_rows,
+    read_parquet,
+    read_images,
+    read_tfrecords,
+)
+
+# api里边要能给concurrency hints
+
+
+def _init_with_boot_policy(
+    config: RuntimeConfigInput | None = None,
+    *,
+    node_mode: str | None = None,
+    node_address: str | None = None,
+    node_id: str | None = None,
+    node_threads: int | None = None,
+    connect_timeout: float | None = None,
+):
+    if config is not None:
+        configure_runtime(config)
+
+    from flownet.platform.user.startup import boot_user_runtime
+
+    return boot_user_runtime(
+        node_mode=node_mode,
+        node_address=node_address,
+        node_id=node_id,
+        node_threads=node_threads,
+        connect_timeout=connect_timeout,
+    )
+
+
+_DEFAULT_NODE_PORT = 8787
+
+
+def _normalize_node_address(node: str | None) -> str | None:
+    if node is None:
+        return None
+    normalized = node.strip()
+    if not normalized:
+        raise ValueError("node address must not be empty.")
+    if ":" in normalized:
+        return normalized
+    return f"{normalized}:{_DEFAULT_NODE_PORT}"
+
+
+def _coerce_user_runtime_config(config: RuntimeConfigInput | None) -> FlownetConfig:
+    if config is None:
+        from flownet.runtime import runtime as runtime_module
+        resolved = coerce_flownet_config(runtime_module._runtime_config)
+    else:
+        resolved = coerce_flownet_config(config)
+    if resolved.schedulable is False:
+        return resolved
+    return replace(resolved, schedulable=False)
+
+
+def init(
+    config: RuntimeConfigInput | None = None,
+    *,
+    node_mode: str | None = None,
+    node_address: str | None = None,
+    node_id: str | None = None,
+    node_threads: int | None = None,
+    connect_timeout: float | None = None,
+):
+    # Compatibility entrypoint:
+    # - preserves legacy behavior where startup mode defaults to env/default.
+    # - for cluster/user decoupled attach flow, prefer init_user(...).
+    return _init_with_boot_policy(
+        config,
+        node_mode=node_mode,
+        node_address=node_address,
+        node_id=node_id,
+        node_threads=node_threads,
+        connect_timeout=connect_timeout,
+    )
+
+
+def init_user(
+    config: RuntimeConfigInput | None = None,
+    *,
+    node: str | None = None,
+    node_address: str | None = None,
+    node_id: str | None = None,
+    node_threads: int | None = None,
+    connect_timeout: float | None = None,
+    node_mode: str = "connect_only",
+):
+    if node is not None and node_address is not None:
+        raise ValueError("Pass either `node` or `node_address`, not both.")
+    normalized_node_address = _normalize_node_address(
+        node if node is not None else node_address
+    )
+    user_runtime_config = _coerce_user_runtime_config(config)
+    # Decoupled user-runtime entrypoint:
+    # - defaults to connect_only so cluster/node lifecycle is managed externally.
+    # - user runtime is never a schedulable compute member in placement plans.
+    # - supports explicit mode override for migration/debug.
+    return _init_with_boot_policy(
+        user_runtime_config,
+        node_mode=node_mode,
+        node_address=normalized_node_address,
+        node_id=node_id,
+        node_threads=node_threads,
+        connect_timeout=connect_timeout,
+    )
+
+__all__ = [
+    "flow",
+    "init",
+    "init_user",
+    "create_actor",
+    "create_local_actor",
+    "list_runtime_topics",
+    "list_runtime_actors",
+    "list_runtime_nodes",
+    "flow_program_ref",
+    "register_flow_program",
+    "list_flow_programs",
+    "get_flow_program",
+    "delete_flow_program",
+    "submit_flow_program",
+    "exception_handler",
+    "flow_exception_handler",
+    "configure_runtime",
+    "lockfree",
+    "topic_stream",
+    "from_topic",
+    "from_service",
+    "TopicStream",
+    "services",
+    "tasks",
+    "node",
+    "node_control",
+    "scheduler",
+    "runtime",
+    "get_services",
+    "try_get_services",
+    "start_service",
+    "publish_service",
+    "stop_service",
+    "list_services",
+    "start_source",
+    "publish_topic",
+    "stop_source",
+    "list_sources",
+    "stateless",
+    "stateless_op",
+    "StatelessOpRef",
+    "StatelessOpConfig",
+    "StatelessScope",
+    "Dataset",
+    "DatasetExecutionMetrics",
+    "DatasetWriteResult",
+    "SINK_IDEMPOTENCY_META_FIELD",
+    "read_rows",
+    "read_parquet",
+    "read_images",
+    "read_tfrecords",
+]

fluty-0.1.1/src/flownet/api/actor_method_ref.py

@@ -0,0 +1,20 @@
+from __future__ import annotations
+
+# ARCH_NOTE:
+# - ActorMethodRef is a concrete execution target (address + actor_id + method),
+#   not a logical alias.
+# - This means flow topology currently captures materialized runtime bindings.
+# - If we later add URI/logical-flow indirection, that resolution should happen
+#   before dispatch and still produce ActorMethodRef for execution.
+class ActorMethodRef:
+    def __init__(self, address: str, actor_id: str, method: str):
+        self.address = address
+        self.actor_id = actor_id
+        self.method = method
+
+    def __call__(self, *args, **kwargs):
+        from flownet.api.runtime import handle_actor_call
+        return handle_actor_call(self, *args, **kwargs)
+    
+    def __reduce__(self):
+        return (self.__class__, (self.address, self.actor_id, self.method))

fluty-0.1.1/src/flownet/api/actor_ref.py

@@ -0,0 +1,30 @@
+from __future__ import annotations
+from dataclasses import dataclass
+from typing import Any, TYPE_CHECKING
+if TYPE_CHECKING:
+    from flownet.api.actor_method_ref import ActorMethodRef
+
+
+class ActorRef:
+    def __init__(self, address: str, actor_id: str):
+        self.address = address
+        self.actor_id = actor_id
+    
+    
+    # TODO: 这里需要区分一下本地actor，并且对于本地actor，直接通过队列调用actor方法并返回结果
+    def __getattr__(self, name: str) -> 'ActorMethodRef':
+        from flownet.api.actor_method_ref import ActorMethodRef
+        return ActorMethodRef(address=self.address, actor_id=self.actor_id, method=name)
+
+    def __reduce__(self):
+        return (self.__class__, (self.address, self.actor_id))
+
+
+class ContainerRef(ActorRef):
+    def __init__(self, address: str):
+        super().__init__(address=address, actor_id="__container__")
+    
+    def __getattr__(self, name):
+        from flownet.api.actor_method_ref import ActorMethodRef
+        print(f"[ContainerRef] Getting method {name} from container at {self.address}")
+        return ActorMethodRef(address=self.address, actor_id="__container__", method=name)