tangle-cli 0.1.16__tar.gz → 0.1.17__tar.gz

{tangle_cli-0.1.16 → tangle_cli-0.1.17}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: tangle-cli
-Version: 0.1.16
+Version: 0.1.17
 Summary: CLI for Tangle, the open-source ML pipeline orchestration platform
 Author: Alexey Volkov, Tangle authors, Volv Grebennikov
 Author-email: Alexey Volkov <alexey.volkov@ark-kun.com>, Volv Grebennikov <volv.grebennikov@shopify.com>
@@ -640,6 +640,61 @@ def score(...):
     ...
 ```
 
+Declare the environment in a config file instead with `TaskEnv.from_config(path)`. A relative `path` resolves against the **calling pipeline module**, never the working directory, so the same script selects the same config wherever `tangle` runs from:
+
+```python
+EVAL = TaskEnv.from_config("envs.yaml")  # relative to THIS file
+```
+
+```yaml
+# envs.yaml
+image: python:3.12
+dependencies_from: pyproject.toml  # relative to THIS config file
+```
+
+The file is read with the same loader `--config` uses, so the `_select` directive works here identically — the case is chosen exactly and case-sensitively by an environment variable, and an unset or unmatched variable is a hard error unless an explicit `default` branch is authored (no implicit default, no implicit production):
+
+```yaml
+_select:
+  env: DEPLOY_ENVIRONMENT
+  cases:
+    production:
+      image: registry.example/scoring:prod
+    staging:
+      image: registry.example/scoring:staging
+  default:
+    image: python:3.12
+```
+
+The document must resolve to exactly one object, and its keys must be named parameters of the generated `__init__` of the class `from_config` was called on — `InitVar` pseudo-fields included, `ClassVar`s and `field(init=False)` excluded — so these files stay environment-only: pipeline concerns such as file paths, versioning, annotations, schedules, or subscriptions have no field to land in and are rejected with the allowed field names. `from_config` is generic — any `TaskEnv` dataclass subclass inherits it unchanged, returns its own type, and is validated against its own fields:
+
+```python
+from tangle_cli.python_pipeline import TaskEnv
+
+@dataclass(frozen=True)
+class GpuEnv(TaskEnv):
+    accelerator: str = ""
+
+    def __post_init__(self):
+        super().__post_init__()
+        if self.accelerator not in ("gpu", "tpu"):
+            raise ValueError("GpuEnv.accelerator must be one of: gpu, tpu")
+
+GPU = GpuEnv.from_config("envs.yaml")  # accepts image, dependencies_from, accelerator
+```
+
+Failures raise `CompileError` naming the resolved config path. A config file is untrusted input, so **no diagnostic echoes a config value**. Keys are rendered through a capped, control-character-scrubbing renderer, and a constructor's own validation text is never quoted — it could embed a value directly, nested inside a structure, or transformed (lower cased, sliced, re-encoded). The rejected exception is also kept off `__cause__`/`__context__`, since `traceback.format_exception` would otherwise print it into the same CI log. The message names the class and the fields present, and points at constructing the class directly to see the validation error:
+
+```
+GpuEnv.from_config: /repo/envs.yaml case is not a valid GpuEnv (fields present:
+accelerator, image). Its validation message is withheld because it can contain
+config values; construct GpuEnv(...) directly to see it.
+```
+
+Calling `GpuEnv(...)` directly is unaffected and raises the ordinary `ValueError` with its full message.
+
+See `examples/python_pipeline/task_env_from_config/` for a runnable example.
+
 Use `@task(image_id="eval-slim")` when source should carry a logical image name instead of a concrete registry reference. Downstream code can register defaults with `register_image_id(...)`, and callers can override at compile time with repeatable `--image ID=REF`:
 
 ```bash

{tangle_cli-0.1.16 → tangle_cli-0.1.17}/README.md

@@ -614,6 +614,61 @@ def score(...):
     ...
 ```
 
+Declare the environment in a config file instead with `TaskEnv.from_config(path)`. A relative `path` resolves against the **calling pipeline module**, never the working directory, so the same script selects the same config wherever `tangle` runs from:
+
+```python
+EVAL = TaskEnv.from_config("envs.yaml")  # relative to THIS file
+```
+
+```yaml
+# envs.yaml
+image: python:3.12
+dependencies_from: pyproject.toml  # relative to THIS config file
+```
+
+The file is read with the same loader `--config` uses, so the `_select` directive works here identically — the case is chosen exactly and case-sensitively by an environment variable, and an unset or unmatched variable is a hard error unless an explicit `default` branch is authored (no implicit default, no implicit production):
+
+```yaml
+_select:
+  env: DEPLOY_ENVIRONMENT
+  cases:
+    production:
+      image: registry.example/scoring:prod
+    staging:
+      image: registry.example/scoring:staging
+  default:
+    image: python:3.12
+```
+
+The document must resolve to exactly one object, and its keys must be named parameters of the generated `__init__` of the class `from_config` was called on — `InitVar` pseudo-fields included, `ClassVar`s and `field(init=False)` excluded — so these files stay environment-only: pipeline concerns such as file paths, versioning, annotations, schedules, or subscriptions have no field to land in and are rejected with the allowed field names. `from_config` is generic — any `TaskEnv` dataclass subclass inherits it unchanged, returns its own type, and is validated against its own fields:
+
+```python
+from tangle_cli.python_pipeline import TaskEnv
+
+@dataclass(frozen=True)
+class GpuEnv(TaskEnv):
+    accelerator: str = ""
+
+    def __post_init__(self):
+        super().__post_init__()
+        if self.accelerator not in ("gpu", "tpu"):
+            raise ValueError("GpuEnv.accelerator must be one of: gpu, tpu")
+
+GPU = GpuEnv.from_config("envs.yaml")  # accepts image, dependencies_from, accelerator
+```
+
+Failures raise `CompileError` naming the resolved config path. A config file is untrusted input, so **no diagnostic echoes a config value**. Keys are rendered through a capped, control-character-scrubbing renderer, and a constructor's own validation text is never quoted — it could embed a value directly, nested inside a structure, or transformed (lower cased, sliced, re-encoded). The rejected exception is also kept off `__cause__`/`__context__`, since `traceback.format_exception` would otherwise print it into the same CI log. The message names the class and the fields present, and points at constructing the class directly to see the validation error:
+
+```
+GpuEnv.from_config: /repo/envs.yaml case is not a valid GpuEnv (fields present:
+accelerator, image). Its validation message is withheld because it can contain
+config values; construct GpuEnv(...) directly to see it.
+```
+
+Calling `GpuEnv(...)` directly is unaffected and raises the ordinary `ValueError` with its full message.
+
+See `examples/python_pipeline/task_env_from_config/` for a runnable example.
+
 Use `@task(image_id="eval-slim")` when source should carry a logical image name instead of a concrete registry reference. Downstream code can register defaults with `register_image_id(...)`, and callers can override at compile time with repeatable `--image ID=REF`:
 
 ```bash

{tangle_cli-0.1.16 → tangle_cli-0.1.17}/packages/tangle-cli/src/tangle_cli/__init__.py

@@ -14,6 +14,6 @@ from tangle_cli.dynamic_discovery_client import TangleDynamicDiscoveryClient
 try:
     __version__ = metadata_version("tangle-cli")
 except PackageNotFoundError:
-    __version__ = "0.1.16"
+    __version__ = "0.1.17"
 
 __all__ = ["TangleDynamicDiscoveryClient", "__version__"]

{tangle_cli-0.1.16 → tangle_cli-0.1.17}/packages/tangle-cli/src/tangle_cli/args_container.py

@@ -41,10 +41,17 @@ class ConfigFileError(Exception):
     """Raised when there is an error loading or resolving a config file."""
 
 
-def _render_case_key(key: str) -> str:
-    """Render an already-validated configured case key for diagnostics."""
+def _render_config_key(key: Any) -> str:
+    """Render a config-document key safely for a diagnostic message.
 
-    rendered = "".join(char if char.isprintable() else "?" for char in key)
+    Config keys come from user files, so a diagnostic must never echo them
+    verbatim: non-printable characters are replaced and the result is length
+    capped before being quoted. Accepts non-string keys (YAML permits them) so
+    callers can report a bad key without first proving it is a string.
+    """
+
+    text = key if isinstance(key, str) else str(key)
+    rendered = "".join(char if char.isprintable() else "?" for char in text)
     if len(rendered) > _MAX_RENDERED_CASE_KEY_LENGTH:
         rendered = rendered[: _MAX_RENDERED_CASE_KEY_LENGTH - 3] + "..."
     return repr(rendered)
@@ -172,8 +179,14 @@ class ArgsContainer:
             if seen is not None and depth <= seen[0]:
                 return seen[2]
 
+        # Every key reaching a diagnostic goes through the ONE capped/scrubbed
+        # renderer: these keys are user input, and an unbounded repr() would
+        # let a hostile document paste control characters or a wall of text
+        # into a compile/CI log.
         siblings = sorted(
-            repr(key) for key in node if not (isinstance(key, str) and key.startswith("_"))
+            _render_config_key(key)
+            for key in node
+            if not (isinstance(key, str) and key.startswith("_"))
         )
         if siblings:
             raise ConfigFileError(
@@ -189,7 +202,9 @@ class ArgsContainer:
         selector_dict = cast(dict[Any, Any], selector)
 
         unexpected = sorted(
-            repr(key) for key in selector_dict if key not in ("env", "cases", "default")
+            _render_config_key(key)
+            for key in selector_dict
+            if key not in ("env", "cases", "default")
         )
         if unexpected:
             raise ConfigFileError(
@@ -231,7 +246,7 @@ class ArgsContainer:
             # Every branch document is shape-checked, not just the selected one,
             # so a malformed selector fails identically in every environment.
             ArgsContainer._validate_branch_document(
-                case_value, f"{SELECT_KEY} case {_render_case_key(case_key)}", depth, memo
+                case_value, f"{SELECT_KEY} case {_render_config_key(case_key)}", depth, memo
             )
 
         default_branch: Any = None
@@ -241,7 +256,7 @@ class ArgsContainer:
                 default_branch, f"{SELECT_KEY}.default", depth, memo
             )
 
-        allowed = ", ".join(_render_case_key(key) for key in sorted(cases_dict))
+        allowed = ", ".join(_render_config_key(key) for key in sorted(cases_dict))
         summary: _SelectorSummary = (env_name, cases_dict, default_branch, allowed)
         if memo is not None:
             previous = memo.get(node_id)

tangle_cli-0.1.17/packages/tangle-cli/src/tangle_cli/python_pipeline/task_env.py

@@ -0,0 +1,203 @@
+"""``TaskEnv`` — declare image + dependencies once, reuse across ``@task``s.
+
+A :class:`TaskEnv` bundles the container ``image`` and an optional
+``dependencies_from`` file so a pipeline author can declare the execution
+environment once and reference it from many ``@task`` components via
+``@task(env=...)``. It is the Python equivalent of a ``local_from_python``
+YAML anchor.
+
+``TaskEnv`` is **authoring-only**. ``@task(env=...)`` expands it at decoration
+time into the existing ``CallableRef._task_image`` /
+``CallableRef._task_dependencies_from`` metadata, so the compiler, hydrator,
+and runner never see a ``TaskEnv`` object.
+
+Example::
+
+    from pathlib import Path
+    from tangle_cli.python_pipeline import TaskEnv, task
+
+    TRAINING = TaskEnv(
+        image="python:3.12",
+        dependencies_from=Path(__file__).parent / "pyproject.toml",
+    )
+
+    @task(env=TRAINING)
+    def train_model(...):
+        ...
+
+An environment can also live in a config file, loaded with
+:meth:`TaskEnv.from_config`, which any ``TaskEnv`` dataclass subclass
+inherits unchanged::
+
+    ENV = TaskEnv.from_config("tangle/envs.yaml")  # relative to THIS file
+"""
+from __future__ import annotations
+
+import inspect
+from dataclasses import dataclass
+from pathlib import Path
+from typing import TypeVar
+
+_TaskEnvT = TypeVar("_TaskEnvT", bound="TaskEnv")
+
+
+@dataclass(frozen=True)
+class TaskEnv:
+    """Reusable execution environment for ``@task`` components.
+
+    Attributes:
+        image: Container image for the component. Required.
+        dependencies_from: Optional path to a ``pyproject.toml`` (or any file
+            the hydrator understands) declaring pip dependencies. A relative
+            path is resolved at the ``TaskEnv`` *definition site*, so a shared
+            ``_envs.py`` resolves intuitively; pass an absolute ``Path`` to
+            avoid that. When omitted, the hydrator's existing dependency
+            discovery still applies.
+    """
+
+    image: str
+    dependencies_from: str | Path | None = None
+
+    def __post_init__(self) -> None:
+        if not isinstance(self.image, str) or not self.image:
+            raise ValueError("TaskEnv.image must be a non-empty string")
+        if self.dependencies_from is None:
+            return
+
+        p = Path(self.dependencies_from)
+        if not p.is_absolute():
+            # Frames: __post_init__ -> generated __init__ -> the definition site.
+            frame = inspect.currentframe()
+            caller = (
+                frame.f_back.f_back
+                if frame and frame.f_back and frame.f_back.f_back
+                else None
+            )
+            filename = caller.f_globals.get("__file__") if caller else None
+            caller_dir = Path(filename).resolve().parent if filename else Path.cwd()
+            p = caller_dir / p
+        # Frozen dataclass: bypass __setattr__ to store the resolved Path.
+        object.__setattr__(self, "dependencies_from", p.resolve())
+
+    @classmethod
+    def from_config(cls: type[_TaskEnvT], path: str | Path) -> _TaskEnvT:
+        """Build the env from a config file, returning an instance of ``cls``.
+
+        Every ``TaskEnv`` dataclass subclass inherits this and is validated
+        against its own fields; nothing here is specific to one kind of
+        environment.
+
+        A relative ``path`` resolves against the **calling file's** directory,
+        never the working directory, so a pipeline script selects the same
+        config wherever the CLI is run from. Call this directly on the class —
+        a subclass that wraps or delegates to it is not supported.
+
+        The file is read with the loader ``--config`` uses, so its ``_select``
+        directive behaves identically here, and it must resolve to exactly one
+        object. The object's keys must be named parameters of ``cls``'s
+        ``__init__``, which keeps an environment config environment-only:
+        pipeline concerns such as versioning or schedules have no field to land
+        in. A relative ``dependencies_from`` inside the object is anchored to
+        the config file's directory rather than to this module.
+
+        Raises:
+            CompileError: for every failure — a missing file, an unresolvable
+                relative path, a load or ``_select`` error, a multi-object
+                document, an unknown or missing field, or a value the subclass
+                rejects.
+
+        Note:
+            A config file is untrusted input, so no diagnostic raised here
+            echoes a config *value* — not in the message, and not through
+            ``__cause__``/``__context__``, which a rendered traceback would
+            print into the same CI log. Construct ``cls(...)`` directly to see
+            a constructor's own validation message.
+        """
+
+        # Lazy import: the authoring surface should not pay for the YAML/CLI
+        # config stack unless a pipeline actually loads an env from a file.
+        from tangle_cli.args_container import (
+            SELECT_KEY,
+            ArgsContainer,
+            ConfigFileError,
+            _render_config_key,
+        )
+        from tangle_cli.python_pipeline.errors import CompileError
+
+        label = f"{cls.__name__}.from_config"
+        config_path = Path(path)
+        if not config_path.is_absolute():
+            frame = inspect.currentframe()
+            caller = frame.f_back if frame is not None else None
+            filename = caller.f_globals.get("__file__") if caller is not None else None
+            if not isinstance(filename, str) or not filename:
+                raise CompileError(
+                    f"{label}({str(path)!r}): a relative path resolves against the "
+                    "calling file, which could not be determined here — pass an "
+                    "absolute path."
+                )
+            config_path = (Path(filename).resolve().parent / config_path).resolve()
+        if not config_path.exists():
+            raise CompileError(
+                f"{label}: {config_path} does not exist. Declare the environment "
+                f"there (optionally a {SELECT_KEY} over an environment variable, "
+                f"with a default case for local runs), or construct "
+                f"{cls.__name__}(...) directly."
+            )
+
+        try:
+            documents = ArgsContainer._load_config_file(config_path)
+        except ConfigFileError as exc:
+            raise CompileError(f"{label} could not resolve {config_path}: {exc}") from exc
+        if len(documents) != 1 or not isinstance(documents[0], dict):
+            raise CompileError(
+                f"{label}: {config_path} must resolve to ONE environment object "
+                f"(one {SELECT_KEY} case), got a multi-config document."
+            )
+
+        case = dict(documents[0])
+        # The signature, not dataclasses.fields(): it is what cls(**case)
+        # actually accepts — InitVar pseudo-fields in, ClassVar and
+        # field(init=False) out. *args/**kwargs are excluded so a typo stays
+        # fail-closed.
+        parameters = {
+            name: parameter
+            for name, parameter in inspect.signature(cls).parameters.items()
+            if parameter.kind
+            in (parameter.POSITIONAL_OR_KEYWORD, parameter.KEYWORD_ONLY)
+        }
+        unknown = sorted(_render_config_key(key) for key in set(case) - set(parameters))
+        if unknown:
+            raise CompileError(
+                f"{label}: {config_path} case has unknown field(s) "
+                f"{', '.join(unknown)}. Allowed fields: {', '.join(parameters)}."
+            )
+        missing = sorted(
+            name
+            for name, parameter in parameters.items()
+            if parameter.default is parameter.empty and name not in case
+        )
+        if missing:
+            raise CompileError(
+                f"{label}: {config_path} case is missing required field(s): "
+                f"{', '.join(missing)}."
+            )
+
+        dependencies = case.get("dependencies_from")
+        if isinstance(dependencies, (str, Path)) and not Path(dependencies).is_absolute():
+            case["dependencies_from"] = (config_path.parent / Path(dependencies)).resolve()
+
+        try:
+            return cls(**case)
+        except (TypeError, ValueError):
+            # Re-raised below, OUTSIDE this handler: the constructor's message
+            # may quote a config value, and `from exc` would publish it on
+            # __cause__ while a raise in-handler would publish it on
+            # __context__ — both get printed by traceback.format_exception.
+            pass
+        raise CompileError(
+            f"{label}: {config_path} case is not a valid {cls.__name__} "
+            f"(fields present: {', '.join(sorted(case))}). Its validation "
+            f"message is withheld because it can contain config values; "
+            f"construct {cls.__name__}(...) directly to see it."
+        )

{tangle_cli-0.1.16 → tangle_cli-0.1.17}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "tangle-cli"
-version = "0.1.16"
+version = "0.1.17"
 description = "CLI for Tangle, the open-source ML pipeline orchestration platform"
 readme = "README.md"
 requires-python = ">=3.10"

{tangle_cli-0.1.16 → tangle_cli-0.1.17}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "tangle-cli"
-version = "0.1.16"
+version = "0.1.17"
 description = "CLI for Tangle, the open-source ML pipeline orchestration platform"
 readme = "README.md"
 authors = [

tangle_cli-0.1.16/packages/tangle-cli/src/tangle_cli/python_pipeline/task_env.py

@@ -1,88 +0,0 @@
-"""``TaskEnv`` — declare image + dependencies once, reuse across ``@task``s.
-
-A :class:`TaskEnv` bundles the container ``image`` and an optional
-``dependencies_from`` file so a pipeline author can declare the execution
-environment once and reference it from many ``@task`` components via
-``@task(env=...)``. It is the Python equivalent of a ``local_from_python``
-YAML anchor.
-
-``TaskEnv`` is **authoring-only**. ``@task(env=...)`` expands it at
-decoration time into the existing ``CallableRef._task_image`` /
-``CallableRef._task_dependencies_from`` metadata. The compiler, hydrator,
-and downstream runner never see a ``TaskEnv`` object — no downstream
-component learns the word ``env``.
-
-Example::
-
-    from pathlib import Path
-    from tangle_cli.python_pipeline import TaskEnv, task
-
-    TRAINING = TaskEnv(
-        image="python:3.12",
-        dependencies_from=Path(__file__).parent / "pyproject.toml",
-    )
-
-    @task(env=TRAINING)
-    def train_model(...):
-        # docstring carries: Metadata / Name: Train Model
-        ...
-
-The component name comes from the function's docstring ``Metadata: Name:``
-block (auto-derived from the function name if absent); the pipeline block name
-(task id) comes from the call-site variable name, or ``.named("Block Name")``
-for an explicit label.
-"""
-from __future__ import annotations
-
-import inspect
-from dataclasses import dataclass
-from pathlib import Path
-
-
-@dataclass(frozen=True)
-class TaskEnv:
-    """Reusable execution environment for ``@task`` components.
-
-    Bundles the container image and optional dependencies file so a
-    pipeline author can declare them once and reference the env from many
-    tasks. This is the Python equivalent of a ``local_from_python`` YAML
-    anchor.
-
-    Attributes:
-        image: Container image for the component. Required — naming the
-            image once is the main point of ``TaskEnv``.
-        dependencies_from: Optional path to a ``pyproject.toml`` (or any
-            file the hydrator understands) declaring pip
-            dependencies. A relative path is resolved at the ``TaskEnv``
-            *definition site* (the module where ``TaskEnv(...)`` is
-            written), so a shared ``_envs.py`` resolves intuitively.
-            Authors can pass an absolute ``Path`` to avoid frame-based
-            ambiguity. When omitted, the existing hydrator/generator
-            dependency discovery still applies.
-    """
-
-    image: str
-    dependencies_from: str | Path | None = None
-
-    def __post_init__(self) -> None:
-        if not isinstance(self.image, str) or not self.image:
-            raise ValueError("TaskEnv.image must be a non-empty string")
-        if self.dependencies_from is None:
-            return
-
-        p = Path(self.dependencies_from)
-        if not p.is_absolute():
-            # Resolve a relative ``dependencies_from`` at the TaskEnv
-            # DEFINITION SITE. Walk frames: __post_init__ -> generated
-            # dataclass __init__ -> the caller that wrote ``TaskEnv(...)``.
-            frame = inspect.currentframe()
-            caller = (
-                frame.f_back.f_back
-                if frame and frame.f_back and frame.f_back.f_back
-                else None
-            )
-            filename = caller.f_globals.get("__file__") if caller else None
-            caller_dir = Path(filename).resolve().parent if filename else Path.cwd()
-            p = caller_dir / p
-        # Frozen dataclass: bypass __setattr__ to store the resolved Path.
-        object.__setattr__(self, "dependencies_from", p.resolve())