pyoco 0.5.0__tar.gz → 0.6.0__tar.gz

{pyoco-0.5.0 → pyoco-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pyoco
-Version: 0.5.0
+Version: 0.6.0
 Summary: A workflow engine with sugar syntax
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
@@ -133,22 +133,29 @@ Or via CLI flag:
 pyoco run --non-cute ...
 ```
 
-## 🔭 Observability Bridge (v0.5)
+## 🔭 Observability / Server (Archived)
 
-- `/metrics` exposes Prometheus counters (`pyoco_runs_total`, `pyoco_runs_in_progress`) and histograms (`pyoco_task_duration_seconds`, `pyoco_run_duration_seconds`). Point Grafana/Prometheus at it to watch pipelines without opening sockets.
-- `/runs` now accepts `status`, `flow`, `limit` query params; `/runs/{id}/logs?tail=100` fetches only the latest snippets for dashboards.
-- Webhook notifications fire when runs COMPLETE/FAIL—configure via `PYOCO_WEBHOOK_*` env vars and forward to Slack or your alerting stack.
-- Import `docs/grafana_pyoco_cute.json` for a lavender/orange starter dashboard (3 panels: in-progress count, completion trend, per-flow latency).
-- 詳細な手順は [docs/observability.md](docs/observability.md) を参照してください。
+Observability and server-related docs are archived and out of scope for the current requirements.  
+See `docs/archive/observability.md` and `docs/archive/roadmap.md`.
 
 ## 🧩 Plug-ins
 
-Need to share domain-specific tasks? Publish an entry point under `pyoco.tasks` and pyoco will auto-load it. See [docs/plugins.md](docs/plugins.md) for the `PluginRegistry` decorator, example `pyproject.toml`, and `pyoco plugins list` CLI helper.
+Need to share domain-specific tasks? Publish an entry point under `pyoco.tasks` and pyoco will auto-load it. We recommend **Task subclasses first** (callables still work with warnings). See [docs/plugins.md](docs/plugins.md) for examples, quickstart, and `pyoco plugins list` / `pyoco plugins lint`.
+
+**Big data note:** pass handles, not copies. For large tensors/images, stash paths or handles in `ctx.artifacts`/`ctx.scratch` and let downstream tasks materialize only when needed. For lazy pipelines (e.g., DataPipe), log the pipeline when you actually iterate (typically the training task) instead of materializing upstream.
+
+## 🧭 Task Discovery (Security)
+
+Pyoco does not allow configuring discovery scope in `flow.yaml` (the `discovery:` key is rejected) to reduce the risk of importing unexpected code.
+
+- **Entry point plug-ins**: auto-loaded from `importlib.metadata.entry_points(group="pyoco.tasks")`
+- **Extra imports (ops-controlled)**: set `PYOCO_DISCOVERY_MODULES` (comma/space-separated module names), e.g. `PYOCO_DISCOVERY_MODULES=tasks,myapp.extra_tasks`
+- **Explicit tasks**: prefer `tasks.<name>.callable` in `flow.yaml` (see tutorials)
 
 ## 📚 Documentation
 
 - [Tutorials](docs/tutorial/index.md)
-- [Roadmap](docs/roadmap.md)
+- [Roadmap (Archived)](docs/archive/roadmap.md)
 
 ## 💖 Contributing
 

{pyoco-0.5.0 → pyoco-0.6.0}/README.md

@@ -121,22 +121,29 @@ Or via CLI flag:
 pyoco run --non-cute ...
 ```
 
-## 🔭 Observability Bridge (v0.5)
+## 🔭 Observability / Server (Archived)
 
-- `/metrics` exposes Prometheus counters (`pyoco_runs_total`, `pyoco_runs_in_progress`) and histograms (`pyoco_task_duration_seconds`, `pyoco_run_duration_seconds`). Point Grafana/Prometheus at it to watch pipelines without opening sockets.
-- `/runs` now accepts `status`, `flow`, `limit` query params; `/runs/{id}/logs?tail=100` fetches only the latest snippets for dashboards.
-- Webhook notifications fire when runs COMPLETE/FAIL—configure via `PYOCO_WEBHOOK_*` env vars and forward to Slack or your alerting stack.
-- Import `docs/grafana_pyoco_cute.json` for a lavender/orange starter dashboard (3 panels: in-progress count, completion trend, per-flow latency).
-- 詳細な手順は [docs/observability.md](docs/observability.md) を参照してください。
+Observability and server-related docs are archived and out of scope for the current requirements.  
+See `docs/archive/observability.md` and `docs/archive/roadmap.md`.
 
 ## 🧩 Plug-ins
 
-Need to share domain-specific tasks? Publish an entry point under `pyoco.tasks` and pyoco will auto-load it. See [docs/plugins.md](docs/plugins.md) for the `PluginRegistry` decorator, example `pyproject.toml`, and `pyoco plugins list` CLI helper.
+Need to share domain-specific tasks? Publish an entry point under `pyoco.tasks` and pyoco will auto-load it. We recommend **Task subclasses first** (callables still work with warnings). See [docs/plugins.md](docs/plugins.md) for examples, quickstart, and `pyoco plugins list` / `pyoco plugins lint`.
+
+**Big data note:** pass handles, not copies. For large tensors/images, stash paths or handles in `ctx.artifacts`/`ctx.scratch` and let downstream tasks materialize only when needed. For lazy pipelines (e.g., DataPipe), log the pipeline when you actually iterate (typically the training task) instead of materializing upstream.
+
+## 🧭 Task Discovery (Security)
+
+Pyoco does not allow configuring discovery scope in `flow.yaml` (the `discovery:` key is rejected) to reduce the risk of importing unexpected code.
+
+- **Entry point plug-ins**: auto-loaded from `importlib.metadata.entry_points(group="pyoco.tasks")`
+- **Extra imports (ops-controlled)**: set `PYOCO_DISCOVERY_MODULES` (comma/space-separated module names), e.g. `PYOCO_DISCOVERY_MODULES=tasks,myapp.extra_tasks`
+- **Explicit tasks**: prefer `tasks.<name>.callable` in `flow.yaml` (see tutorials)
 
 ## 📚 Documentation
 
 - [Tutorials](docs/tutorial/index.md)
-- [Roadmap](docs/roadmap.md)
+- [Roadmap (Archived)](docs/archive/roadmap.md)
 
 ## 💖 Contributing
 

{pyoco-0.5.0 → pyoco-0.6.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "pyoco"
-version = "0.5.0"
+version = "0.6.0"
 description = "A workflow engine with sugar syntax"
 readme = "README.md"
 requires-python = ">=3.10"

{pyoco-0.5.0 → pyoco-0.6.0}/src/pyoco/__init__.py

@@ -2,10 +2,11 @@ from .core.models import Flow, Task
 from .core.engine import Engine
 from .dsl.syntax import task
 from .trace.console import ConsoleTraceBackend
+from . import support
 
 def run(flow: Flow, params: dict = None, trace: bool = True, cute: bool = True):
     backend = ConsoleTraceBackend(style="cute" if cute else "plain")
     engine = Engine(trace_backend=backend)
     return engine.run(flow, params)
 
-__all__ = ["task", "Flow", "run"]
+__all__ = ["task", "Flow", "run", "support"]

{pyoco-0.5.0 → pyoco-0.6.0}/src/pyoco/cli/main.py

@@ -4,13 +4,22 @@ import sys
 import os
 import signal
 import time
+from types import SimpleNamespace
 from ..schemas.config import PyocoConfig
 from ..discovery.loader import TaskLoader
 from ..core.models import Flow
 from ..core.engine import Engine
 from ..trace.console import ConsoleTraceBackend
 from ..client import Client
-from ..discovery.plugins import list_available_plugins
+from ..support.service import SupportInfoService
+from ..core.exceptions import (
+    SupportInfoError,
+    InvalidFormatError,
+    TaskNotFoundError,
+    InvalidFilterError,
+    OutputWriteError,
+    MissingTaskMetadataError,
+)
 
 def main():
     parser = argparse.ArgumentParser(description="Pyoco Workflow Engine")
@@ -88,6 +97,32 @@ def main():
     plugins_sub = plugins_parser.add_subparsers(dest="plugins_command")
     plugins_list = plugins_sub.add_parser("list", help="List discovered plug-ins")
     plugins_list.add_argument("--json", action="store_true", help="Output JSON payload")
+    plugins_lint = plugins_sub.add_parser("lint", help="Validate plug-ins for upcoming requirements")
+    plugins_lint.add_argument("--json", action="store_true", help="Output JSON payload")
+
+    support_parser = subparsers.add_parser("support", help="Generate support info")
+    support_subparsers = support_parser.add_subparsers(dest="support_command")
+
+    support_tasks = support_subparsers.add_parser("tasks", help="List tasks for LLM support")
+    support_tasks.add_argument("--config", required=True, help="Path to flow.yaml")
+    support_tasks.add_argument("--format", default="prompt", choices=["prompt", "json", "md"])
+    support_tasks.add_argument("--output", help="Write output to file")
+    support_tasks.add_argument("--name", action="append", help="Filter by task name (repeatable)")
+    support_tasks.add_argument("--origin", action="append", help="Filter by origin (repeatable)")
+    support_tasks.add_argument("--tag", action="append", help="Filter by tag (repeatable)")
+
+    support_task = support_subparsers.add_parser("task", help="Show task detail for LLM support")
+    support_task.add_argument("--config", required=True, help="Path to flow.yaml")
+    support_task.add_argument("--name", required=True, help="Task name")
+    support_task.add_argument("--format", default="prompt", choices=["prompt", "json", "md"])
+    support_task.add_argument("--output", help="Write output to file")
+    support_task.add_argument("--origin", action="append", help="Filter by origin (repeatable)")
+    support_task.add_argument("--tag", action="append", help="Filter by tag (repeatable)")
+
+    support_guide = support_subparsers.add_parser("guide", help="Show flow.yaml guide for LLM support")
+    support_guide.add_argument("--config", required=True, help="Path to flow.yaml")
+    support_guide.add_argument("--format", default="prompt", choices=["prompt", "json", "md"])
+    support_guide.add_argument("--output", help="Write output to file")
 
     args = parser.parse_args()
 
@@ -120,18 +155,72 @@ def main():
         return
 
     if args.command == "plugins":
-        infos = list_available_plugins()
+        reports = _collect_plugin_reports()
         if args.plugins_command == "list":
             if getattr(args, "json", False):
-                print(json.dumps(infos, indent=2))
+                print(json.dumps(reports, indent=2))
             else:
-                if not infos:
+                if not reports:
                     print("No plug-ins registered under group 'pyoco.tasks'.")
                 else:
                     print("Discovered plug-ins:")
-                    for info in infos:
+                    for info in reports:
                         mod = info.get("module") or info.get("value")
                         print(f" - {info.get('name')} ({mod})")
+                        if info.get("error"):
+                            print(f"     ⚠️  error: {info['error']}")
+                            continue
+                        for task in info.get("tasks", []):
+                            warn_msg = "; ".join(task.get("warnings", [])) or "ok"
+                            print(f"     • {task['name']} [{task['origin']}] ({warn_msg})")
+                        for warn in info.get("warnings", []):
+                            print(f"     ⚠️  {warn}")
+        elif args.plugins_command == "lint":
+            issues = []
+            for info in reports:
+                prefix = info["name"]
+                if info.get("error"):
+                    issues.append(f"{prefix}: {info['error']}")
+                    continue
+                for warn in info.get("warnings", []):
+                    issues.append(f"{prefix}: {warn}")
+                for task in info.get("tasks", []):
+                    for warn in task.get("warnings", []):
+                        issues.append(f"{prefix}.{task['name']}: {warn}")
+            payload = {"issues": issues, "reports": reports}
+            if getattr(args, "json", False):
+                print(json.dumps(payload, indent=2))
+            else:
+                if not issues:
+                    print("✅ All plug-ins look good.")
+                else:
+                    print("⚠️  Plug-in issues found:")
+                    for issue in issues:
+                        print(f" - {issue}")
+            if issues:
+                sys.exit(1)
+        else:
+            plugins_parser.print_help()
+        return
+
+    if args.command == "support":
+        if not args.support_command:
+            support_parser.print_help()
+            sys.exit(1)
+        filters = _build_support_filters(args)
+        try:
+            content = SupportInfoService().build(
+                kind=args.support_command,
+                config_path=args.config,
+                format=args.format,
+                filters=filters or None,
+                output_path=args.output,
+            )
+        except SupportInfoError as exc:
+            _print_support_error(exc)
+            sys.exit(1)
+        if not args.output:
+            print(content)
         return
 
     if args.command == "server":
@@ -368,6 +457,45 @@ def main():
             sys.exit(2 if args.dry_run else 1)
         return
 
+def _collect_plugin_reports():
+    dummy = SimpleNamespace(tasks={})
+    loader = TaskLoader(dummy)
+    loader.load()
+    return loader.plugin_reports
+
+
+def _build_support_filters(args):
+    filters = {}
+    if getattr(args, "name", None):
+        value = args.name
+        filters["name"] = value if isinstance(value, list) else [value]
+    if getattr(args, "origin", None):
+        filters["origin"] = args.origin
+    if getattr(args, "tag", None):
+        filters["tag"] = args.tag
+    return filters
+
+
+def _print_support_error(exc: SupportInfoError) -> None:
+    if isinstance(exc, InvalidFormatError):
+        print(f"Invalid format: {exc.format}")
+        return
+    if isinstance(exc, TaskNotFoundError):
+        print(f"Task not found: {exc.name}")
+        return
+    if isinstance(exc, OutputWriteError):
+        print(f"Failed to write output: {exc.path}")
+        return
+    if isinstance(exc, InvalidFilterError):
+        print(f"Invalid filter: {exc.filter_value}")
+        return
+    if isinstance(exc, MissingTaskMetadataError):
+        fields = ",".join(exc.fields)
+        print(f"Missing task metadata: {exc.name} fields={fields}")
+        return
+    print(f"Error: {exc}")
+
+
 def _stream_logs(client, args):
     seen_seq = -1
     follow = args.follow

pyoco-0.6.0/src/pyoco/core/exceptions.py

@@ -0,0 +1,51 @@
+class ControlFlowError(Exception):
+    """Base error for control flow execution issues."""
+
+
+class UntilMaxIterationsExceeded(ControlFlowError):
+    def __init__(self, expression: str, max_iter: int):
+        super().__init__(f"Until condition '{expression}' exceeded max_iter={max_iter}")
+        self.expression = expression
+        self.max_iter = max_iter
+
+
+class SwitchNoMatch(ControlFlowError):
+    def __init__(self, expression: str):
+        super().__init__(f"Switch expression '{expression}' did not match any case.")
+        self.expression = expression
+
+
+class SupportInfoError(Exception):
+    """Base error for support info generation."""
+
+
+class InvalidFormatError(SupportInfoError):
+    def __init__(self, format: str):
+        self.format = format
+        super().__init__(f"Invalid format: {format}")
+
+
+class TaskNotFoundError(SupportInfoError):
+    def __init__(self, name: str):
+        self.name = name
+        super().__init__(f"Task not found: {name}")
+
+
+class InvalidFilterError(SupportInfoError):
+    def __init__(self, filter_value: str):
+        self.filter_value = filter_value
+        super().__init__(f"Invalid filter: {filter_value}")
+
+
+class OutputWriteError(SupportInfoError):
+    def __init__(self, path: str):
+        self.path = path
+        super().__init__(f"Failed to write output: {path}")
+
+
+class MissingTaskMetadataError(SupportInfoError):
+    def __init__(self, name: str, fields: list[str]):
+        self.name = name
+        self.fields = fields
+        field_list = ",".join(fields)
+        super().__init__(f"Missing task metadata: {name} fields={field_list}")

{pyoco-0.5.0 → pyoco-0.6.0}/src/pyoco/core/models.py

@@ -238,11 +238,10 @@ class Flow:
             # So `flow >> (A | B)` just adds A and B.
             # Then `(A | B) >> C` is handled by Branch.
             pass
-        
-        # Update tail
+
         if new_tasks:
             self._tail = set(new_tasks)
-            
+
         return self
 
     def add_task(self, task: Task):
@@ -284,3 +283,45 @@ class Flow:
             tail_task.dependents.add(task)
             task.dependencies.add(tail_task)
         self._tail = {task}
+
+
+@dataclass
+class TaskIO:
+    name: str
+    type: str
+    required: bool
+    constraints: Optional[List[str]] = None
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "TaskIO":
+        return cls(
+            name=data.get("name"),
+            type=data.get("type"),
+            required=data.get("required"),
+            constraints=data.get("constraints"),
+        )
+
+
+@dataclass
+class TaskInfo:
+    name: str
+    summary: str
+    inputs: List[TaskIO]
+    outputs: List[TaskIO]
+    origin: Optional[str] = None
+    tags: Optional[List[str]] = None
+
+
+@dataclass
+class SupportFilters:
+    name: Optional[List[str]] = None
+    origin: Optional[List[str]] = None
+    tag: Optional[List[str]] = None
+
+
+@dataclass
+class SupportInfo:
+    kind: str
+    format: str
+    content: str
+    filters: SupportFilters

{pyoco-0.5.0 → pyoco-0.6.0}/src/pyoco/discovery/loader.py

@@ -1,6 +1,5 @@
 import importlib
-import pkgutil
-import sys
+import os
 from typing import Dict, List, Any, Set
 from ..core.models import Task
 from ..dsl.syntax import TaskWrapper
@@ -11,30 +10,28 @@ class TaskLoader:
         self.config = config
         self.strict = strict
         self.tasks: Dict[str, Task] = {}
+        self.task_infos: Dict[str, Any] = {}
         self._explicit_tasks: Set[str] = set()
         self.plugin_reports: List[Dict[str, Any]] = []
 
     def load(self):
         # Load explicitly defined tasks in config FIRST (Higher priority)
         for task_name, task_conf in self.config.tasks.items():
-            if task_conf.callable:
-                self._load_explicit_task(task_name, task_conf)
+            callable_path = self._conf_get(task_conf, "callable")
+            if callable_path:
+                self._load_explicit_task(task_name, task_conf, callable_path)
                 self._explicit_tasks.add(task_name)
 
-        # Load from packages
-        for package in self.config.discovery.packages:
-            self._load_package(package)
-        
-        # Load from entry points (simplified)
-        for ep in self.config.discovery.entry_points:
-            self._load_module(ep)
-            
-        # Load from glob modules
-        for pattern in self.config.discovery.glob_modules:
-            self._load_glob_modules(pattern)
+        self._load_env_modules()
 
         self._load_entry_point_plugins()
 
+    def _load_env_modules(self) -> None:
+        raw = os.getenv("PYOCO_DISCOVERY_MODULES", "")
+        modules = [item.strip() for item in raw.replace(",", " ").split() if item.strip()]
+        for module_name in modules:
+            self._load_module(module_name)
+
     def _register_task(self, name: str, task: Task):
         if name in self.tasks:
             if name in self._explicit_tasks:
@@ -51,24 +48,27 @@ class TaskLoader:
         # Apply config overlay if exists
         if self.config and name in self.config.tasks:
             conf = self.config.tasks[name]
-            if not conf.callable:
-                if conf.inputs:
-                    task.inputs.update(conf.inputs)
-                if conf.outputs:
-                    task.outputs.extend(conf.outputs)
+            if not self._conf_get(conf, "callable"):
+                inputs = self._conf_get(conf, "inputs") or {}
+                outputs = self._conf_get(conf, "outputs") or []
+                if inputs:
+                    task.inputs.update(inputs)
+                if outputs:
+                    task.outputs.extend(outputs)
 
         self.tasks[name] = task
 
-    def _load_package(self, package_name: str):
-        try:
-            pkg = importlib.import_module(package_name)
-            if hasattr(pkg, '__path__'):
-                for _, name, _ in pkgutil.iter_modules(pkg.__path__, pkg.__name__ + "."):
-                    self._load_module(name)
+    def _register_task_info(self, info: Any):
+        name = getattr(info, "name", None)
+        if not name:
+            return
+        if name in self.task_infos:
+            msg = f"Task metadata '{name}' already defined."
+            if self.strict:
+                raise ValueError(f"{msg} (Strict mode enabled)")
             else:
-                self._scan_module(pkg)
-        except ImportError as e:
-            print(f"Warning: Could not import package {package_name}: {e}")
+                print(f"Warning: {msg} Overwriting.")
+        self.task_infos[name] = info
 
     def _load_module(self, module_name: str):
         try:
@@ -76,30 +76,6 @@ class TaskLoader:
             self._scan_module(mod)
         except ImportError as e:
             print(f"Warning: Could not import module {module_name}: {e}")
-            
-    def _load_glob_modules(self, pattern: str):
-        import glob
-        import os
-        
-        # Pattern is likely a file path glob, e.g. "jobs/*.py"
-        # We need to convert file paths to module paths
-        files = glob.glob(pattern, recursive=True)
-        for file_path in files:
-            if not file_path.endswith(".py"):
-                continue
-                
-            # Convert path to module
-            # This is tricky without knowing the root. 
-            # Assumption: running from root, and file path is relative to root.
-            # e.g. "myproject/tasks/foo.py" -> "myproject.tasks.foo"
-            
-            rel_path = os.path.relpath(file_path)
-            if rel_path.startswith(".."):
-                # Out of tree, skip or warn
-                continue
-                
-            module_name = rel_path.replace(os.sep, ".")[:-3] # strip .py
-            self._load_module(module_name)
 
     def _load_entry_point_plugins(self):
         entries = iter_entry_points()
@@ -109,6 +85,7 @@ class TaskLoader:
                 "value": ep.value,
                 "module": getattr(ep, "module", ""),
                 "tasks": [],
+                "warnings": [],
             }
             registry = PluginRegistry(self, ep.name)
             try:
@@ -116,7 +93,11 @@ class TaskLoader:
                 if not callable(hook):
                     raise TypeError("Entry point must be callable")
                 hook(registry)
-                info["tasks"] = list(registry.registered_names)
+                info["tasks"] = list(registry.records)
+                info["task_infos"] = list(registry.task_infos.values())
+                info["warnings"] = list(registry.warnings)
+                if not registry.records:
+                    info["warnings"].append("no tasks registered")
             except Exception as exc:
                 info["error"] = str(exc)
                 if self.strict:
@@ -130,13 +111,17 @@ class TaskLoader:
                 self._register_task(name, obj.task)
             elif isinstance(obj, Task):
                 self._register_task(name, obj)
-            elif callable(obj) and getattr(obj, '__pyoco_task__', False):
-                # Convert to Task if not already
-                pass
 
-    def _load_explicit_task(self, name: str, conf: Any):
+    def _conf_get(self, conf: Any, key: str):
+        if hasattr(conf, key):
+            return getattr(conf, key)
+        if isinstance(conf, dict):
+            return conf.get(key)
+        return None
+
+    def _load_explicit_task(self, name: str, conf: Any, callable_path: str):
         # Load callable
-        module_path, func_name = conf.callable.split(':')
+        module_path, func_name = callable_path.split(':')
         try:
             mod = importlib.import_module(module_path)
             obj = getattr(mod, func_name)
@@ -150,8 +135,8 @@ class TaskLoader:
                 
             # Create a Task wrapper
             t = Task(func=real_func, name=name)
-            t.inputs = conf.inputs
-            t.outputs = conf.outputs
+            t.inputs = self._conf_get(conf, "inputs") or {}
+            t.outputs = self._conf_get(conf, "outputs") or []
             self.tasks[name] = t
         except (ImportError, AttributeError) as e:
             print(f"Error loading task {name}: {e}")