lely 0.2.0__tar.gz → 0.3.0__tar.gz

{lely-0.2.0 → lely-0.3.0}/CHANGELOG.md

@@ -6,6 +6,27 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.3.0] - 2026-10-08
+
+A breaking release: the `command` plugin is gone, and a step runs on its own.
+
+### Added
+
+- **A step on its own** (`spec/011`). A step that inherits `lely.step.Step` and ends with
+  `Step.main()` is a command of its own: `plan`, `apply`, `destroy`, `status` and `check`,
+  its options as flags, with no `lely.yml`. A plan it writes is a one-step lely plan:
+  `lely apply plan.json` takes it, and the same checks are made before anything runs.
+  `check` holds the step to the rules from the command line; `--live` applies and destroys
+  on the target.
+- **`Program`**: a program as a step, in ten lines — one `run` every apply, destructive if
+  it says so, outputs from the JSON on its last line. What `command` did, under the rules.
+
+### Removed
+
+- **`command`**: a program is a `Program` (ten lines, see Writing a step); a config that
+  names `command` is told by `lely validate` that no plugin of that name is installed.
+  → [011](https://github.com/kostavo-oss/lely/blob/main/spec/011-a-step-on-its-own.md)
+
 ## [0.2.0] - 2026-10-07
 
 ### Added

{lely-0.2.0 → lely-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: lely
-Version: 0.2.0
+Version: 0.3.0
 Summary: One plan for your whole Databricks deploy: the bundle and the steps around it, reviewed before anything runs
 Project-URL: Homepage, https://github.com/kostavo-oss/lely
 Project-URL: Documentation, https://kostavo-oss.github.io/lely/
@@ -118,23 +118,24 @@ steps:
         model_version: ${steps.model.version}     # … and the bundle says what it takes
 
   - name: backfill               # needs something from the bundle: it is below it …
-    uses: command
-    with:
-      apply: [./ops/backfill.sh, "${steps.app.resources.jobs.backfill.id}"]   # … and says what
+    uses: ./ops/steps.py:Backfill
+    with: {job: "${steps.app.resources.jobs.backfill.id}"}   # … and says what
 ```
 
 A step takes a value from another step in one way only: `${steps.<name>.<output>}`, in its own
 options, and only from a step above it. Reading a step's `with:` is enough to know everything
 it depends on.
 
-`bundle` and `command` come with lely. `./ops/steps.py:LatestModel` is a plugin of your own —
-a class in a file in the repo. This one runs as it stands; your lookup goes where the `14` is:
+`bundle` comes with lely; the other two are yours — classes in a file in the repo.
+`LatestModel` runs as it stands, and your lookup goes where the `14` is; `Backfill` is a
+program as a step, one run every apply:
 
 ```python
 # ops/steps.py
 from dataclasses import dataclass
 
 from lely.model import Output, StepPlan
+from lely.step import Program
 
 
 class LatestModel:
@@ -154,6 +155,17 @@ class LatestModel:
 
     def apply(self, ctx, plan):
         return plan.outputs  # nothing to do: what it gives was known at plan
+
+
+class Backfill(Program):
+    """Backfills the job this deploy made."""
+
+    @dataclass(frozen=True, slots=True)
+    class Options:
+        job: str
+
+    def command(self, ctx):
+        return ["./ops/backfill.sh", ctx.options.job]
 ```
 
 The rest is yours too: a bundle beside `lely.yml` with a variable `model_version` and a job
@@ -187,7 +199,7 @@ lely plan · target dev · https://dbc-example.cloud.databricks.com as jane@exam
     ± pipelines.foo  destructive
         replaced: storage (immutable)
     ▶ uploads the bundle's files
-  notify  command
+  notify  ./ops/steps.py:Notify
     ⏸ waiting for app.resources.jobs.bar.id
   backfill  bundle.run
     bundle  ← app
@@ -254,17 +266,15 @@ what they can do.
 - **`bundle`** — an Asset Bundle: planned, deployed, listed and destroyed through the
   Databricks CLI. It gives the bundle's variables, its resources' names, and their ids and
   links once they exist.
-- **`command`** — commands you give it: `apply`, and optionally `plan`, `destroy` and the
-  `outputs` it gives. No shell, and no secrets in arguments.
 - **`bundle.run`** — runs a job, pipeline or app from a bundle step.
 - **Your own** — a class in a file in your repo (`uses: ./ops/steps.py:LatestModel`), or a
-  package that registers one under the `lely.steps` entry point. `lely.testing` checks it
-  against the same rules as the ones above.
+  package that registers one under the `lely.steps` entry point. A program is a `Program`,
+  in ten lines. `lely.testing` checks it against the same rules as the ones above.
 - **`stevin`** — tables, planned by [stevin](https://github.com/kostavo-oss/stevin). Parked:
   it can plan, and can't apply yet.
 
-Planning runs your project's own code — a plugin in the repo, a `command` step's plan command.
-On a pull request, give `lely plan` credentials that can read and nothing more.
+Planning runs your project's own code — a step in the repo. On a pull request, give
+`lely plan` credentials that can read and nothing more.
 
 ## What has been tried
 

{lely-0.2.0 → lely-0.3.0}/README.md

@@ -87,23 +87,24 @@ steps:
         model_version: ${steps.model.version}     # … and the bundle says what it takes
 
   - name: backfill               # needs something from the bundle: it is below it …
-    uses: command
-    with:
-      apply: [./ops/backfill.sh, "${steps.app.resources.jobs.backfill.id}"]   # … and says what
+    uses: ./ops/steps.py:Backfill
+    with: {job: "${steps.app.resources.jobs.backfill.id}"}   # … and says what
 ```
 
 A step takes a value from another step in one way only: `${steps.<name>.<output>}`, in its own
 options, and only from a step above it. Reading a step's `with:` is enough to know everything
 it depends on.
 
-`bundle` and `command` come with lely. `./ops/steps.py:LatestModel` is a plugin of your own —
-a class in a file in the repo. This one runs as it stands; your lookup goes where the `14` is:
+`bundle` comes with lely; the other two are yours — classes in a file in the repo.
+`LatestModel` runs as it stands, and your lookup goes where the `14` is; `Backfill` is a
+program as a step, one run every apply:
 
 ```python
 # ops/steps.py
 from dataclasses import dataclass
 
 from lely.model import Output, StepPlan
+from lely.step import Program
 
 
 class LatestModel:
@@ -123,6 +124,17 @@ class LatestModel:
 
     def apply(self, ctx, plan):
         return plan.outputs  # nothing to do: what it gives was known at plan
+
+
+class Backfill(Program):
+    """Backfills the job this deploy made."""
+
+    @dataclass(frozen=True, slots=True)
+    class Options:
+        job: str
+
+    def command(self, ctx):
+        return ["./ops/backfill.sh", ctx.options.job]
 ```
 
 The rest is yours too: a bundle beside `lely.yml` with a variable `model_version` and a job
@@ -156,7 +168,7 @@ lely plan · target dev · https://dbc-example.cloud.databricks.com as jane@exam
     ± pipelines.foo  destructive
         replaced: storage (immutable)
     ▶ uploads the bundle's files
-  notify  command
+  notify  ./ops/steps.py:Notify
     ⏸ waiting for app.resources.jobs.bar.id
   backfill  bundle.run
     bundle  ← app
@@ -223,17 +235,15 @@ what they can do.
 - **`bundle`** — an Asset Bundle: planned, deployed, listed and destroyed through the
   Databricks CLI. It gives the bundle's variables, its resources' names, and their ids and
   links once they exist.
-- **`command`** — commands you give it: `apply`, and optionally `plan`, `destroy` and the
-  `outputs` it gives. No shell, and no secrets in arguments.
 - **`bundle.run`** — runs a job, pipeline or app from a bundle step.
 - **Your own** — a class in a file in your repo (`uses: ./ops/steps.py:LatestModel`), or a
-  package that registers one under the `lely.steps` entry point. `lely.testing` checks it
-  against the same rules as the ones above.
+  package that registers one under the `lely.steps` entry point. A program is a `Program`,
+  in ten lines. `lely.testing` checks it against the same rules as the ones above.
 - **`stevin`** — tables, planned by [stevin](https://github.com/kostavo-oss/stevin). Parked:
   it can plan, and can't apply yet.
 
-Planning runs your project's own code — a plugin in the repo, a `command` step's plan command.
-On a pull request, give `lely plan` credentials that can read and nothing more.
+Planning runs your project's own code — a step in the repo. On a pull request, give
+`lely plan` credentials that can read and nothing more.
 
 ## What has been tried
 

{lely-0.2.0 → lely-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "lely"
-version = "0.2.0"
+version = "0.3.0"
 description = "One plan for your whole Databricks deploy: the bundle and the steps around it, reviewed before anything runs"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -47,7 +47,6 @@ lely = "lely.cli:main"
 # The plugins lely ships register exactly as anyone's would.
 [project.entry-points."lely.steps"]
 bundle = "lely.steps.bundle:Bundle"
-command = "lely.steps.command:Command"
 stevin = "lely.steps.stevin:Stevin"
 "bundle.run" = "lely.steps.bundle_run:BundleRun"
 

{lely-0.2.0 → lely-0.3.0}/src/lely/cli.py

@@ -957,9 +957,8 @@ def doctor(path: ConfigOption = None, profile: ProfileOption = None) -> None:
             line(None, f"what these credentials may do couldn't be read: {error}")
     line(
         None,
-        "planning runs the project's own code — a plugin in the repo, a `command` "
-        "step's plan command. On a pull request, give `lely plan` credentials that "
-        "can read and nothing more.",
+        "planning runs the project's own code — a step in the repo. On a pull "
+        "request, give `lely plan` credentials that can read and nothing more.",
     )
     if loaded is None:
         line(None, f"no project checked: {unread}")

lely-0.3.0/src/lely/flags.py

@@ -0,0 +1,236 @@
+"""A step's `Options` as the flags of its own command line, and back into a
+config block — for a step run on its own (`spec/011`, R5).
+
+Each field of the `Options` dataclass is `--<name>`, with the field's type, its
+default and the `#:` comment above it as its help. What a flag gives is then
+written into the same config nodes a `lely.yml` would hold, so the step's
+options are read by `lely.options` exactly as under lely: one reader, one set
+of errors, one `made_from` hash.
+
+The types a flag can carry: `str`, `int`, `float`, `bool` (as a word: `true`,
+`yes`, `no`…, so a bundle variable can set it), `Secret` (text), `Literal[…]`
+(one of them), `tuple[T, …]` (the flag more than once), `Mapping[str, T]`
+(`--name key=value`, more than once), and `T | None`. Anything else is refused
+when the step starts, with this list.
+"""
+
+from __future__ import annotations
+
+import dataclasses
+import types
+import typing
+from collections.abc import Mapping, Sequence
+from dataclasses import dataclass
+from typing import TYPE_CHECKING, Any, Literal
+
+from typer.core import TyperOption
+
+from lely.config import Entry, Loc, Map, Scalar, Seq
+from lely.errors import LelyError, Refused
+from lely.model import Linked, Secret
+from lely.options import default_of, option_fields
+from lely.registry import option_docs
+
+if TYPE_CHECKING:
+    from typer._click.types import ParamType
+else:  # typer carries its own click since 0.21
+    try:
+        from typer._click.types import ParamType
+    except ImportError:  # pragma: no cover - typer before its own click
+        from click import ParamType
+
+#: Where an option given as a flag "was written".
+COMMAND_LINE = Loc("command line", 0, 0)
+
+_SCALARS = (str, int, float, bool)
+
+
+class _YesNo(ParamType):
+    """A yes or a no as a job's parameter can give it: a word, not a flag."""
+
+    name = "true|false"
+
+    def convert(self, value: Any, param: Any, ctx: Any) -> bool:
+        if isinstance(value, bool):
+            return value
+        word = str(value).strip().lower()
+        if word in ("true", "1", "yes", "y"):
+            return True
+        if word in ("false", "0", "no", "n"):
+            return False
+        self.fail(f"'{value}' is not one of true, 1, yes, y, false, 0, no, n", param, ctx)
+
+
+class _OneOf(ParamType):
+    """One of a `Literal`'s values, of its own kind."""
+
+    def __init__(self, allowed: tuple[Any, ...]) -> None:
+        self.allowed = allowed
+        self.name = "|".join(str(a) for a in allowed)
+
+    def convert(self, value: Any, param: Any, ctx: Any) -> Any:
+        for allowed in self.allowed:
+            if value == allowed or (isinstance(value, str) and str(allowed) == value):
+                return allowed
+        self.fail(f"'{value}' is not one of {self.name}", param, ctx)
+
+
+class _Pair(ParamType):
+    """`key=value`, for a mapping option."""
+
+    name = "key=value"
+
+    def convert(self, value: Any, param: Any, ctx: Any) -> tuple[str, str]:
+        if isinstance(value, tuple):
+            return value
+        key, separator, rest = str(value).partition("=")
+        if not separator or not key.strip():
+            self.fail(f"'{value}' is not key=value", param, ctx)
+        return key.strip(), rest
+
+
+@dataclass(frozen=True, slots=True)
+class Flag:
+    """One option of the step, as its flag takes it."""
+
+    field: str
+    name: str
+    kind: Any
+    many: bool
+    required: bool
+    default: Any
+    help: str
+
+    @property
+    def option(self) -> str:
+        return "--" + self.field.replace("_", "-")
+
+
+def flags_of(options: type) -> tuple[Flag, ...]:
+    """The flags for an `Options` class. Raises `LelyError` for a field no
+    flag can carry."""
+    hints = typing.get_type_hints(options)
+    docs = option_docs(options)
+    result: list[Flag] = []
+    for f in option_fields(options):
+        made = default_of(options, f)
+        required = made is dataclasses.MISSING
+        kind, many = _kind(hints[f.name], f.name, options)
+        result.append(
+            Flag(
+                field=f.name,
+                name=f.name.replace("_", "-"),
+                kind=kind,
+                many=many,
+                required=required,
+                default=None if required else made,
+                help=docs.get(f.name, ""),
+            )
+        )
+    return tuple(result)
+
+
+def _kind(tp: Any, name: str, options: type) -> tuple[Any, bool]:
+    """What a flag for a field of type `tp` takes, and whether more than once."""
+    origin = typing.get_origin(tp)
+    args = typing.get_args(tp)
+    if origin in (types.UnionType, typing.Union):
+        members = [a for a in args if a is not type(None)]
+        if len(members) == 1:
+            return _kind(members[0], name, options)
+    elif origin is Literal:
+        return _OneOf(args), False
+    elif origin in (tuple, list):
+        element = args[0] if args else str
+        if element is Ellipsis or element in _SCALARS or element is Secret:
+            inner, _ = _kind(str if element is Ellipsis else element, name, options)
+            return inner, True
+    elif origin in (dict, Mapping) or tp in (dict, Mapping):
+        element = args[1] if len(args) == 2 else str
+        if element in _SCALARS or element is Secret:
+            return _Pair(), True
+    elif tp is bool:
+        return _YesNo(), False
+    elif tp in (str, int, float):
+        return tp, False
+    elif tp is Secret:
+        return str, False
+    which = "the name of a step" if tp is Linked else _said(tp)
+    raise LelyError(
+        f"`{options.__qualname__}.{name}` is {which}: a flag can't carry it. On its "
+        "own, a step takes text, whole and decimal numbers, true or false, a secret, "
+        "one of a few values, a list or a mapping of those, and `T | None`. Run it "
+        "under lely for the rest."
+    )
+
+
+def _said(tp: Any) -> str:
+    return getattr(tp, "__name__", None) or str(tp)
+
+
+def options_of(flags: Sequence[Flag]) -> list[TyperOption]:
+    """The step's own flags, as typer builds a command from."""
+    made = []
+    for flag in flags:
+        kind: Any = flag.kind
+        # No default here: a flag not given is left out of the block, so the
+        # option's own default applies and the step is "made from" the same
+        # thing as a config that doesn't write it. The default is said in the help.
+        help_ = flag.help
+        if not flag.required and not flag.many and flag.default is not None:
+            help_ = f"{help_} [default: {flag.default}]".strip()
+        made.append(
+            TyperOption(
+                param_decls=[flag.option, flag.field],
+                type=kind,
+                required=flag.required,
+                multiple=flag.many,
+                default=None,
+                show_default=False,
+                help=help_ or None,
+            )
+        )
+    return made
+
+
+def block_from(flags: Sequence[Flag], values: Mapping[str, Any]) -> Map:
+    """What the flags gave, as the `with:` block a config would hold.
+
+    A flag not given is left out, so the option's own default applies. A value
+    that looks like a reference is refused: on its own, a step has no step
+    above to take it from (`spec/011`, R8).
+    """
+    entries: list[Entry] = []
+    for flag in flags:
+        value = values.get(flag.field)
+        if value is None or (flag.many and not value):
+            continue
+        _no_reference(flag, value)
+        entries.append(Entry(flag.field, COMMAND_LINE, _node(value, flag)))
+    return Map(tuple(entries), COMMAND_LINE)
+
+
+def _node(value: Any, flag: Flag) -> Scalar | Seq | Map:
+    if flag.many and isinstance(flag.kind, _Pair):
+        pairs = [
+            v if isinstance(v, tuple) else _Pair().convert(v, None, None) for v in value
+        ]
+        return Map(
+            tuple(Entry(k, COMMAND_LINE, Scalar(v, COMMAND_LINE)) for k, v in pairs),
+            COMMAND_LINE,
+        )
+    if flag.many:
+        return Seq(tuple(Scalar(v, COMMAND_LINE) for v in value), COMMAND_LINE)
+    return Scalar(value, COMMAND_LINE)
+
+
+def _no_reference(flag: Flag, value: Any) -> None:
+    texts = [value] if not isinstance(value, list | tuple) else list(value)
+    for text in texts:
+        if isinstance(text, tuple):
+            text = text[1]
+        if isinstance(text, str) and "${" in text:
+            raise Refused(
+                f"{flag.option} is `{text}`: on its own, a step has no step above to "
+                "take a `${…}` reference from. Run it under lely, or pass the value."
+            )

{lely-0.2.0 → lely-0.3.0}/src/lely/planfile.py

@@ -11,9 +11,10 @@ A secret never reaches the file: a secret output is written as
 holding one is refused outright. Nothing in the file is kept for apply — every
 step is planned again, and gives its values again.
 
-A step's plan has the same shape here as a `command` step's plan command
-prints, so `step_plan_from_json` reads both. What a run leaves behind — a
-`Result`, a `Status` — is written here too, and never read back.
+A step's plan is written by `step_plan_to_json` and read back by
+`step_plan_from_json`; `lely.testing` holds a plugin's plan to that round trip.
+What a run leaves behind — a `Result`, a `Status` — is written here too, and
+never read back.
 """
 
 from __future__ import annotations
@@ -79,7 +80,7 @@ _CHANGE_KEYS = frozenset({"key", "action", "summary", "destructive", "detail"})
 
 
 class PlanFileError(LelyError):
-    """A plan file (or a plan command's output) that can't be read."""
+    """A plan file that can't be read."""
 
 
 #: The longest view a step's plan keeps, in characters: a view is a picture of

{lely-0.2.0 → lely-0.3.0}/src/lely/process.py

@@ -1,11 +1,11 @@
-"""Running another program: the Databricks CLI, stevin, a `command` step.
+"""Running another program: the Databricks CLI, stevin, a step's program.
 
 One place, so every step that shells out fails the same way: a missing program
 says so, and a failing one is shown in its own words — what it wrote on both of
 its streams.
 
 A program that only answers a question is run to its end and its output taken.
-One that changes something — a deploy, a job run, a step's apply command — can
+One that changes something — a deploy, a job run, a step's program — can
 take minutes, so its caller may ask to hear it (`said`): each line is passed on
 as it comes, and still kept for whoever reads the result.
 """

{lely-0.2.0 → lely-0.3.0}/src/lely/render/markdown.py

@@ -18,7 +18,7 @@
     -     replace  pipelines.ingest  destructive
               replaced: storage (immutable)
           run      uploads the bundle's files
-      notify  command
+      notify  ./ops/steps.py:Notify
           waiting for app.resources.jobs.backfill.id
     ```
 

{lely-0.2.0 → lely-0.3.0}/src/lely/render/rich.py

@@ -10,9 +10,9 @@
         ± pipelines.ingest  destructive
             replaced: storage (immutable)
         ▶ uploads the bundle's files
-      notify  command
+      notify  ./ops/steps.py:Notify
         ⏸ waiting for app.resources.jobs.backfill.id
-      warm  command
+      warm  ./ops/steps.py:Warm
         – skipped: not for target `dev`
 
     Plan: 2 changes · 1 run · 1 destructive · 1 waiting