pipeline-frame 0.0.1__tar.gz

pipeline_frame-0.0.1/PKG-INFO

@@ -0,0 +1,103 @@
+Metadata-Version: 2.4
+Name: pipeline-frame
+Version: 0.0.1
+Summary: A middleware framework for building data processing pipelines
+Author-email: alan <al6nlee@gmail.com>
+Project-URL: Homepage, https://github.com/al6nlee/pipeline-frame
+Project-URL: Bug Reports, https://github.com/al6nlee/pipeline-frame/issues
+Project-URL: Source, https://github.com/al6nlee/pipeline-frame
+Keywords: middleware,pipeline,execution,processor
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: License :: OSI Approved :: MIT License
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+
+# Pipeline Frame
+
+一个用于构建 middleware-style processing pipeline 的轻量 Python 框架。
+每个 `Processor` 分别处理进入和返回阶段的数据，从而形成 U-shaped
+执行路径：
+
+```text
+input:  Processor A -> Processor B -> Processor C
+output: Processor A <- Processor B <- Processor C
+```
+
+适用于 request/response middleware、数据校验与转换链、ETL 和 event
+processing 等场景。应用启动入口创建一个 application-scoped `Engine`，
+由它注册 processor 并构建可复用的 `Workflow`；每次调用 workflow 时，
+再创建一个 one-shot `Execution`。
+
+## 快速开始 / Quick Start
+
+安装方式以及完整的 sync/async 示例，请参阅
+[Quick Start](./docs/quick-start.md)。
+
+## 架构概览 / Architecture
+
+运行时分为两层：application-scoped `Engine` 注册并构建 frozen
+`Workflow`；每次调用 workflow 时创建 one-shot `Execution`，由它拥有本次
+调用的 processor instances、`ExecutionContext`、`ExecutionState` 和
+`ExecutionResult`。完整的 ownership、生命周期、执行顺序和失败契约请参阅
+[Architecture](./docs/architecture.md)。
+
+## 项目结构 / Package Layout
+
+```text
+pipeline_frame/
+├── __init__.py
+├── execution.py
+├── execution_context.py
+├── execution_result.py
+├── execution_state.py
+├── engine.py
+├── processor.py
+├── workflow.py
+└── observability/
+    ├── __init__.py
+    ├── logger.py
+    └── timer.py
+```
+
+## 开发 / Development
+
+在项目根目录运行测试：
+
+```bash
+python3 -m unittest discover -s tests -v
+```
+
+使用标准 Python build frontend 构建 distribution：
+
+```bash
+python3 -m pip install build
+python3 -m build
+```
+
+## 发布 / Release
+
+向 `main` push 包含 `pyproject.toml` 中 `project.version` 变更的提交，
+即视为发布新版本。CI 会在同一次 workflow 中创建 annotated
+`v<version>` tag、构建 distribution 并发布到 PyPI。也支持手动 push
+`v*` tag。
+
+如果之前的 release 已创建 tag，但在发布到 PyPI 前失败，后续 `main`
+push 会在 PyPI 仍缺少对应版本时复用该 tag，无需再次修改 version。
+
+version 可以写成 `0.2.0` 或 `v0.2.0`，CI 会统一生成 `v0.2.0` tag。
+在 `pyproject.toml` 中省略 `v` 是更常规的写法。
+
+启用自动发布前，需要在 GitHub 仓库设置中为 Actions 开启
+**Read and write permissions**，并为该 workflow 配置 PyPI Trusted
+Publishing。每次发布必须使用新版本，PyPI 不允许覆盖已有版本。
+
+## 项目链接 / Project Links
+
+- [Source repository](https://github.com/al6nlee/pipeline-frame)
+- [Issue tracker](https://github.com/al6nlee/pipeline-frame/issues)
+
+## License
+
+MIT

pipeline_frame-0.0.1/README.md

@@ -0,0 +1,87 @@
+# Pipeline Frame
+
+一个用于构建 middleware-style processing pipeline 的轻量 Python 框架。
+每个 `Processor` 分别处理进入和返回阶段的数据，从而形成 U-shaped
+执行路径：
+
+```text
+input:  Processor A -> Processor B -> Processor C
+output: Processor A <- Processor B <- Processor C
+```
+
+适用于 request/response middleware、数据校验与转换链、ETL 和 event
+processing 等场景。应用启动入口创建一个 application-scoped `Engine`，
+由它注册 processor 并构建可复用的 `Workflow`；每次调用 workflow 时，
+再创建一个 one-shot `Execution`。
+
+## 快速开始 / Quick Start
+
+安装方式以及完整的 sync/async 示例，请参阅
+[Quick Start](./docs/quick-start.md)。
+
+## 架构概览 / Architecture
+
+运行时分为两层：application-scoped `Engine` 注册并构建 frozen
+`Workflow`；每次调用 workflow 时创建 one-shot `Execution`，由它拥有本次
+调用的 processor instances、`ExecutionContext`、`ExecutionState` 和
+`ExecutionResult`。完整的 ownership、生命周期、执行顺序和失败契约请参阅
+[Architecture](./docs/architecture.md)。
+
+## 项目结构 / Package Layout
+
+```text
+pipeline_frame/
+├── __init__.py
+├── execution.py
+├── execution_context.py
+├── execution_result.py
+├── execution_state.py
+├── engine.py
+├── processor.py
+├── workflow.py
+└── observability/
+    ├── __init__.py
+    ├── logger.py
+    └── timer.py
+```
+
+## 开发 / Development
+
+在项目根目录运行测试：
+
+```bash
+python3 -m unittest discover -s tests -v
+```
+
+使用标准 Python build frontend 构建 distribution：
+
+```bash
+python3 -m pip install build
+python3 -m build
+```
+
+## 发布 / Release
+
+向 `main` push 包含 `pyproject.toml` 中 `project.version` 变更的提交，
+即视为发布新版本。CI 会在同一次 workflow 中创建 annotated
+`v<version>` tag、构建 distribution 并发布到 PyPI。也支持手动 push
+`v*` tag。
+
+如果之前的 release 已创建 tag，但在发布到 PyPI 前失败，后续 `main`
+push 会在 PyPI 仍缺少对应版本时复用该 tag，无需再次修改 version。
+
+version 可以写成 `0.2.0` 或 `v0.2.0`，CI 会统一生成 `v0.2.0` tag。
+在 `pyproject.toml` 中省略 `v` 是更常规的写法。
+
+启用自动发布前，需要在 GitHub 仓库设置中为 Actions 开启
+**Read and write permissions**，并为该 workflow 配置 PyPI Trusted
+Publishing。每次发布必须使用新版本，PyPI 不允许覆盖已有版本。
+
+## 项目链接 / Project Links
+
+- [Source repository](https://github.com/al6nlee/pipeline-frame)
+- [Issue tracker](https://github.com/al6nlee/pipeline-frame/issues)
+
+## License
+
+MIT

pipeline_frame-0.0.1/pipeline_frame/__init__.py

@@ -0,0 +1,9 @@
+"""Package metadata for Pipeline Frame."""
+
+from importlib.metadata import PackageNotFoundError, version as _distribution_version
+
+try:
+    __version__ = _distribution_version("pipeline-frame")
+except PackageNotFoundError:
+    # Source checkouts are not always installed as a distribution.
+    __version__ = "0.2.0"

pipeline_frame-0.0.1/pipeline_frame/engine.py

@@ -0,0 +1,134 @@
+"""Application-scoped registries for processors and workflows."""
+
+from collections.abc import Iterable, Mapping
+import inspect
+from threading import RLock
+from types import MappingProxyType
+
+from .processor import Processor, _validate_name
+from .workflow import Workflow
+
+
+class Engine:
+    """Registry and workflow factory for one application entrypoint.
+
+    ``Engine`` deliberately has no process-wide singleton semantics. An
+    application can create independent engines for different entrypoints or
+    test fixtures; each instance owns its own processor and workflow registry.
+    Execution is performed by ``Execution`` instances created from workflows.
+    """
+
+    def __init__(self) -> None:
+        self._processors: dict[str, type[Processor]] = {}
+        self._workflows: dict[str, Workflow] = {}
+        self._lock = RLock()
+
+    @property
+    def processors(self) -> Mapping[str, type[Processor]]:
+        """A read-only view of registered processors."""
+
+        return MappingProxyType(self._processors)
+
+    @property
+    def workflows(self) -> Mapping[str, Workflow]:
+        """A read-only view of registered workflows."""
+
+        return MappingProxyType(self._workflows)
+
+    def register_processor(self, processor_cls: type[Processor]) -> type[Processor]:
+        """Register a concrete ``Processor`` subclass.
+
+        The class's ``name`` is its canonical registry identity. Instances are
+        intentionally not accepted: ``Workflow`` must be able to create a fresh
+        instance for every ``Execution``.
+        """
+
+        if not inspect.isclass(processor_cls) or not issubclass(
+            processor_cls, Processor
+        ):
+            raise TypeError("processor_cls must be a Processor subclass")
+        if inspect.isabstract(processor_cls):
+            raise TypeError(
+                f"Processor {processor_cls.__name__!r} must implement input() and output()"
+            )
+        name = _validate_name(processor_cls.name, "processor name")
+
+        with self._lock:
+            if name in self._processors:
+                raise ValueError(f"Processor {name!r} already exists")
+            self._processors[name] = processor_cls
+        return processor_cls
+
+    def register_workflow(self, workflow: Workflow) -> Workflow:
+        """Register an existing immutable workflow under its canonical name."""
+
+        if not isinstance(workflow, Workflow):
+            raise TypeError("workflow must be a Workflow instance")
+        name = _validate_name(workflow.name, "workflow name")
+
+        with self._lock:
+            if name in self._workflows:
+                raise ValueError(f"Workflow {name!r} already exists")
+            self._workflows[name] = workflow
+        return workflow
+
+    def create_workflow(
+        self,
+        name: str,
+        processor_names: Iterable[str],
+    ) -> Workflow:
+        """Resolve registered processor names and register a new workflow.
+
+        ``processor_names`` is consumed immediately and copied into the
+        immutable workflow definition. Repeated names are allowed, which is
+        useful when a processor intentionally appears at multiple stages.
+        """
+
+        _validate_name(name, "workflow name")
+        if isinstance(processor_names, (str, bytes)):
+            raise TypeError("processor_names must be an iterable of names, not a string")
+        try:
+            names = tuple(processor_names)
+        except TypeError as exc:
+            raise TypeError("processor_names must be an iterable of names") from exc
+
+        with self._lock:
+            if name in self._workflows:
+                raise ValueError(f"Workflow {name!r} already exists")
+
+            definitions: list[type[Processor]] = []
+            for index, processor_name in enumerate(names):
+                if not isinstance(processor_name, str):
+                    raise TypeError(
+                        f"processor name at index {index} must be a string"
+                    )
+                try:
+                    processor_cls = self._processors[processor_name]
+                except KeyError as exc:
+                    raise KeyError(
+                        f"Unknown processor {processor_name!r} while creating "
+                        f"workflow {name!r}"
+                    ) from exc
+                definitions.append(processor_cls)
+
+            workflow = Workflow(name, tuple(definitions))
+            self._workflows[name] = workflow
+            return workflow
+
+    def get_processor(self, name: str) -> type[Processor]:
+        """Return a registered processor class with contextual lookup errors."""
+
+        with self._lock:
+            try:
+                return self._processors[name]
+            except KeyError as exc:
+                raise KeyError(f"Unknown processor {name!r}") from exc
+
+    def get_workflow(self, name: str) -> Workflow:
+        """Return a registered workflow with contextual lookup errors."""
+
+        with self._lock:
+            try:
+                return self._workflows[name]
+            except KeyError as exc:
+                raise KeyError(f"Unknown workflow {name!r}") from exc

pipeline_frame-0.0.1/pipeline_frame/execution.py

@@ -0,0 +1,316 @@
+"""One-shot execution of an immutable workflow definition."""
+
+import asyncio
+import inspect
+import time
+from threading import RLock
+from typing import Any
+
+from .execution_context import ExecutionContext
+from .execution_result import ExecutionResult, ExecutionStatus
+from .execution_state import ExecutionState
+from .observability.logger import logger as default_logger
+from .observability.timer import timer
+
+
+class Execution:
+    """Run a workflow once with fresh processor instances.
+
+    ``Execution`` owns the processor instances and all three per-call
+    runtime objects.  It deliberately has no public ``status`` property;
+    lifecycle information belongs to ``ExecutionResult.status`` while a
+    private admission flag only enforces the one-shot rule.
+    """
+
+    def __init__(self, workflow: Any, processors: Any = None):
+        self.workflow = workflow
+        if processors is None:
+            factory = getattr(workflow, "_create_processor_instances", None)
+            if not callable(factory):
+                raise TypeError("processors are required for this workflow")
+            processors = factory()
+        self.processors = tuple(processors)
+        self.context: ExecutionContext | None = None
+        self.state: ExecutionState | None = None
+        self._result = ExecutionResult()
+
+        self._started = False
+        self._lock = RLock()
+
+    @property
+    def result(self) -> ExecutionResult:
+        """The result accumulator owned by this one-shot execution."""
+
+        return self._result
+
+    # ------------------------------------------------------------------
+    # Public execution entry points
+    # ------------------------------------------------------------------
+    def run(self, input_data: Any) -> ExecutionResult:
+        """Execute synchronously and return a frozen ``ExecutionResult``.
+
+        If a processor returns an awaitable, synchronous execution fails with
+        a clear instruction to use :meth:`arun`; the one-shot admission is
+        still consumed and the resulting failure is frozen.
+        """
+
+        context, state, result = self._begin(input_data)
+        time_report: list[Any] = []
+        run_logger: Any = default_logger
+        total_t0 = time.perf_counter()
+        terminal_status = ExecutionStatus.SUCCESS
+        terminal_error: BaseException | None = None
+
+        try:
+            run_logger = self._select_logger(context)
+            for processor in self.processors:
+                with timer(processor.name, "INPUT", time_report):
+                    self._call_sync(
+                        processor.input,
+                        context,
+                        state,
+                        result,
+                    )
+
+            for processor in reversed(self.processors):
+                with timer(processor.name, "OUTPUT", time_report):
+                    self._call_sync(
+                        processor.output,
+                        context,
+                        state,
+                        result,
+                    )
+
+            return result
+        except asyncio.CancelledError as exc:
+            # A synchronous caller can still deliberately raise
+            # CancelledError; preserve the same cancellation contract as arun.
+            terminal_status = ExecutionStatus.CANCELLED
+            terminal_error = exc
+            raise
+        except BaseException as exc:
+            terminal_status = ExecutionStatus.FAILED
+            terminal_error = exc
+            raise
+        finally:
+            self._finish(
+                result,
+                run_logger,
+                time_report,
+                total_t0,
+                terminal_status,
+                terminal_error,
+            )
+
+    async def arun(self, input_data: Any) -> ExecutionResult:
+        """Execute asynchronously, allowing sync and async hooks to mix."""
+
+        context, state, result = self._begin(input_data)
+        time_report: list[Any] = []
+        run_logger: Any = default_logger
+        total_t0 = time.perf_counter()
+        terminal_status = ExecutionStatus.SUCCESS
+        terminal_error: BaseException | None = None
+
+        try:
+            run_logger = self._select_logger(context)
+            for processor in self.processors:
+                with timer(processor.name, "INPUT", time_report):
+                    await self._call_async(
+                        processor.input,
+                        context,
+                        state,
+                        result,
+                    )
+
+            for processor in reversed(self.processors):
+                with timer(processor.name, "OUTPUT", time_report):
+                    await self._call_async(
+                        processor.output,
+                        context,
+                        state,
+                        result,
+                    )
+
+            return result
+        except asyncio.CancelledError as exc:
+            terminal_status = ExecutionStatus.CANCELLED
+            terminal_error = exc
+            raise
+        except BaseException as exc:
+            terminal_status = ExecutionStatus.FAILED
+            terminal_error = exc
+            raise
+        finally:
+            self._finish(
+                result,
+                run_logger,
+                time_report,
+                total_t0,
+                terminal_status,
+                terminal_error,
+            )
+
+    # ------------------------------------------------------------------
+    # Admission and finalization
+    # ------------------------------------------------------------------
+    def _begin(
+        self, input_data: Any
+    ) -> tuple[ExecutionContext, ExecutionState, ExecutionResult]:
+        """Admit one caller and construct its runtime objects.
+
+        The result is installed before context validation so malformed input
+        still leaves ``execution.result`` as a frozen failure and cannot be
+        retried through the same one-shot object.
+        """
+
+        with self._lock:
+            if self._started:
+                raise RuntimeError("Execution is one-shot and has already started")
+            result = self.result
+            if result.frozen:
+                raise RuntimeError("ExecutionResult is already frozen")
+            # Reserve the accumulator before releasing the admission lock so
+            # an external ``freeze()`` cannot race the one-shot transition.
+            result._begin_execution()
+            self._started = True
+            # Keep one stable result object for the whole one-shot lifecycle.
+            # In particular, callers can retain ``execution.result`` before
+            # calling ``run`` and observe its terminal state afterwards.
+
+        total_t0 = time.perf_counter()
+        run_logger: Any = default_logger
+        time_report: list[Any] = []
+        try:
+            result.set_status(ExecutionStatus.RUNNING)
+            context = ExecutionContext(input_data)
+            state = ExecutionState()
+            self.context = context
+            self.state = state
+            run_logger = self._select_logger(context)
+            return context, state, result
+        except asyncio.CancelledError as exc:
+            self._finish(
+                result,
+                run_logger,
+                time_report,
+                total_t0,
+                ExecutionStatus.CANCELLED,
+                exc,
+            )
+            raise
+        except BaseException as exc:
+            self._finish(
+                result,
+                run_logger,
+                time_report,
+                total_t0,
+                ExecutionStatus.FAILED,
+                exc,
+            )
+            raise
+
+    def _finish(
+        self,
+        result: ExecutionResult,
+        run_logger: Any,
+        time_report: list[Any],
+        total_t0: float,
+        terminal_status: ExecutionStatus,
+        terminal_error: BaseException | None,
+    ) -> None:
+        # Freeze before logging so observers see the same terminal object the
+        # caller receives.  Logger/observer failures must never replace the
+        # processor exception (or turn a successful run into a failure).
+        try:
+            # Timing is kept in a local sink while hooks are running. The
+            # result keeps its active marker until the merge and freeze have
+            # both completed, preventing an external observer from freezing
+            # the result in between those operations.
+            result._finish_execution(
+                time_report,
+                status=terminal_status,
+                error=terminal_error,
+            )
+        except BaseException:
+            # Freezing is internal bookkeeping. A malformed custom report
+            # object must not replace the business exception or strand the
+            # execution in RUNNING.
+            pass
+        finally:
+            try:
+                workflow_name = getattr(self.workflow, "name", "workflow")
+            except BaseException:
+                workflow_name = "workflow"
+            self._safe_log_run(
+                run_logger,
+                result,
+                time_report,
+                total_t0,
+                workflow_name,
+            )
+
+    @staticmethod
+    def _select_logger(context: ExecutionContext) -> Any:
+        try:
+            candidate = context.input.get("logger", default_logger)
+            return (
+                candidate
+                if callable(getattr(candidate, "info", None))
+                else default_logger
+            )
+        except BaseException:
+            return default_logger
+
+    @staticmethod
+    def _safe_log_run(
+        run_logger: Any,
+        result: ExecutionResult,
+        time_report: list[Any],
+        total_t0: float,
+        workflow_name: Any = "workflow",
+    ) -> None:
+        try:
+            status = getattr(result.status, "value", result.status)
+            run_logger.info(f"{workflow_name} execution finished, status: {status}")
+            run_logger.info(
+                f"{workflow_name} execution total time: "
+                f"{time.perf_counter() - total_t0:.2f}s"
+            )
+            for stage, name, cost in time_report:
+                run_logger.info(f"{stage}: {name} elapsed: {cost:.2f}s")
+        except BaseException:
+            return
+
+    # ------------------------------------------------------------------
+    # Processor hook invocation
+    # ------------------------------------------------------------------
+    @staticmethod
+    def _call_sync(
+        action: Any,
+        context: ExecutionContext,
+        state: ExecutionState,
+        result: ExecutionResult,
+    ) -> None:
+        outcome = action(context, state, result)
+        if inspect.isawaitable(outcome):
+            close = getattr(outcome, "close", None)
+            if callable(close):
+                try:
+                    close()
+                except BaseException:
+                    pass
+            raise RuntimeError(
+                "Async processor methods require calling Execution.arun()"
+            )
+
+    @staticmethod
+    async def _call_async(
+        action: Any,
+        context: ExecutionContext,
+        state: ExecutionState,
+        result: ExecutionResult,
+    ) -> None:
+        outcome = action(context, state, result)
+        if inspect.isawaitable(outcome):
+            await outcome

pipeline_frame-0.0.1/pipeline_frame/execution_context.py

@@ -0,0 +1,41 @@
+"""Read-only input context for a single pipeline execution."""
+
+from collections.abc import Iterator, Mapping
+from types import MappingProxyType
+from typing import Any
+
+
+class ExecutionContext(Mapping[str, Any]):
+    """Immutable, per-execution view of the input mapping.
+
+    The mapping is copied at construction time and wrapped in a
+    :class:`~types.MappingProxyType`, so assigning top-level keys through the
+    context cannot alter either the context or the caller's mapping.  Values
+    are intentionally not deep-copied; the first version of the runtime only
+    guarantees top-level read-only semantics.
+    """
+
+    __slots__ = ("_input",)
+
+    def __init__(self, input_data: Mapping[str, Any]):
+        if not isinstance(input_data, Mapping):
+            raise TypeError("ExecutionContext input_data must be a Mapping")
+        self._input = MappingProxyType(dict(input_data))
+
+    @property
+    def input(self) -> Mapping[str, Any]:
+        """The read-only input mapping for this execution."""
+
+        return self._input
+
+    def __getitem__(self, key: str) -> Any:
+        return self._input[key]
+
+    def __iter__(self) -> Iterator[str]:
+        return iter(self._input)
+
+    def __len__(self) -> int:
+        return len(self._input)
+
+    def __repr__(self) -> str:
+        return f"{type(self).__name__}({dict(self._input)!r})"