pyinfra 2.9.2__tar.gz → 3.0b1__tar.gz

pyinfra-3.0b1/CHANGELOG.md

@@ -0,0 +1,29 @@
+# v3.0.beta1
+
+Welcome to pyinfra v3! This version is the biggest overhaul of pyinfra since it was created back in 2015. Most v2 deployment code should be automatically compatible, but as always be aware. Major changes:
+
+## Runtime operation execution
+
+pyinfra now executes operations at runtime, rather than pre-generating commands. Although the change isn't noticeable this fixes an entire class of bugs and confusion. See the [limitations](https://docs.pyinfra.com/en/2.x/deploy-process.html#limitations) section in the v2 docs. All of those issues are now a thing of the past.
+
+This represents a huge overhaul of pyinfra's internals and should be a huge improvement for users.
+
+Care has been taken to reduce the overhead of this change which still supports the same diffs and change proposal mechanism.
+
+## CLI flow & prompts
+
+The pyinfra CLI will now prompt (instead of ignore, or immediately exit) when problems are encountered, allowing the user to choose to continue. Additionally an approval step is added before executing changes (skip with `-y`).
+
+## Extendable connectors API, typing overhaul
+
+v3 of pyinfra includes for the first time a (mostly) typed internal API with proper support for IDE linting. There's a whole new connectors API that provides a framework for building new connectors.
+
+More TBC...
+
+# v2.x
+
+[See this file in the `2.x` branch](https://github.com/Fizzadar/pyinfra/blob/2.x/CHANGELOG.md).
+
+# v1.x
+
+[See this file in the `1.x` branch](https://github.com/Fizzadar/pyinfra/blob/1.x/CHANGELOG.md).

{pyinfra-2.9.2/pyinfra.egg-info → pyinfra-3.0b1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pyinfra
-Version: 2.9.2
+Version: 3.0b1
 Summary: pyinfra automates/provisions/manages/deploys infrastructure.
 Home-page: https://pyinfra.com
 Author: Nick / Fizzadar
@@ -16,7 +16,6 @@ Classifier: Intended Audience :: Information Technology
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.6
 Classifier: Programming Language :: Python :: 3.7
 Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
@@ -28,7 +27,6 @@ Description-Content-Type: text/markdown
 Provides-Extra: test
 Provides-Extra: docs
 Provides-Extra: dev
-Provides-Extra: ansible
 License-File: LICENSE.md
 
 <p align="center">
@@ -37,13 +35,17 @@ License-File: LICENSE.md
     </a>
 </p>
 
-<p align="center">
-    <em>pyinfra automates infrastructure using Python. It’s fast and scales from one server to thousands. Great for ad-hoc command execution, service deployment, configuration management and more.</em>
+<p>
+    <strong>Note: this is the v3 branch, which is currently in beta. <a href="https://docs.pyinfra.com/en/next">See the docs for v3</a>. If needed the <a href="https://github.com/pyinfra-dev/pyinfra/tree/2.x/">2.x branch is here</a>, but is in bugfix only mode.</strong>
+</p>
+
+<p>
+    pyinfra turns Python code into shell commands and runs them on your servers. Execute ad-hoc commands and write declarative operations. Target SSH servers, local machine and Docker containers. Fast and scales from one server to thousands. Think <code>ansible</code> but Python instead of YAML, and a lot faster.
 </p>
 
 ---
 
-<p align="center">
+<p>
     <a href="https://docs.pyinfra.com"><strong>Documentation</strong></a> &rArr;
     <a href="https://docs.pyinfra.com/page/getting-started.html"><strong>Getting Started</strong></a> &bull;
     <a href="https://docs.pyinfra.com/page/examples.html"><strong>Examples</strong></a> &bull;
@@ -51,7 +53,7 @@ License-File: LICENSE.md
     <a href="https://docs.pyinfra.com/page/contributing.html"><strong>Contributing</strong></a>
 </p>
 
-<p align="center">
+<p>
     Chat &rArr;
     <a href="https://matrix.to/#/#pyinfra:matrix.org"><strong><code>#pyinfra</code> on Matrix</strong></a>
 </p>

{pyinfra-2.9.2 → pyinfra-3.0b1}/README.md

@@ -4,13 +4,17 @@
     </a>
 </p>
 
-<p align="center">
-    <em>pyinfra automates infrastructure using Python. It’s fast and scales from one server to thousands. Great for ad-hoc command execution, service deployment, configuration management and more.</em>
+<p>
+    <strong>Note: this is the v3 branch, which is currently in beta. <a href="https://docs.pyinfra.com/en/next">See the docs for v3</a>. If needed the <a href="https://github.com/pyinfra-dev/pyinfra/tree/2.x/">2.x branch is here</a>, but is in bugfix only mode.</strong>
+</p>
+
+<p>
+    pyinfra turns Python code into shell commands and runs them on your servers. Execute ad-hoc commands and write declarative operations. Target SSH servers, local machine and Docker containers. Fast and scales from one server to thousands. Think <code>ansible</code> but Python instead of YAML, and a lot faster.
 </p>
 
 ---
 
-<p align="center">
+<p>
     <a href="https://docs.pyinfra.com"><strong>Documentation</strong></a> &rArr;
     <a href="https://docs.pyinfra.com/page/getting-started.html"><strong>Getting Started</strong></a> &bull;
     <a href="https://docs.pyinfra.com/page/examples.html"><strong>Examples</strong></a> &bull;
@@ -18,7 +22,7 @@
     <a href="https://docs.pyinfra.com/page/contributing.html"><strong>Contributing</strong></a>
 </p>
 
-<p align="center">
+<p>
     Chat &rArr;
     <a href="https://matrix.to/#/#pyinfra:matrix.org"><strong><code>#pyinfra</code> on Matrix</strong></a>
 </p>

{pyinfra-2.9.2 → pyinfra-3.0b1}/pyinfra/api/__init__.py

@@ -11,6 +11,9 @@ from .config import Config  # noqa: F401 # pragma: no cover
 from .deploy import deploy  # noqa: F401 # pragma: no cover
 from .exceptions import DeployError  # noqa: F401 # pragma: no cover
 from .exceptions import (  # noqa: F401
+    FactError,
+    FactTypeError,
+    FactValueError,
     InventoryError,
     OperationError,
     OperationTypeError,

pyinfra-3.0b1/pyinfra/api/arguments.py

@@ -0,0 +1,339 @@
+from __future__ import annotations
+
+from typing import (
+    TYPE_CHECKING,
+    Any,
+    Callable,
+    Generic,
+    Iterable,
+    List,
+    Mapping,
+    Optional,
+    TypeVar,
+    Union,
+    cast,
+    get_type_hints,
+)
+
+from typing_extensions import TypedDict
+
+from pyinfra import context
+from pyinfra.api.exceptions import ArgumentTypeError
+from pyinfra.api.state import State
+from pyinfra.api.util import raise_if_bad_type
+
+if TYPE_CHECKING:
+    from pyinfra.api.config import Config
+    from pyinfra.api.host import Host
+
+T = TypeVar("T")
+default_sentinel = object()
+
+
+class ArgumentMeta(Generic[T]):
+    description: str
+    default: Callable[["Config"], T]
+    handler: Optional[Callable[["Config", T], T]]
+
+    def __init__(self, description, default, handler=None) -> None:
+        self.description = description
+        self.default = default
+        self.handler = handler
+
+
+# Connector arguments
+# These are arguments passed to the various connectors that provide the underlying
+# API to read/write external systems.
+
+
+# Note: ConnectorArguments is specifically not total as it's used to type many
+# functions via Unpack and we don't want to specify every kwarg.
+class ConnectorArguments(TypedDict, total=False):
+    # Auth arguments
+    _sudo: bool
+    _sudo_user: str
+    _use_sudo_login: bool
+    _sudo_password: str
+    _preserve_sudo_env: bool
+    _su_user: str
+    _use_su_login: bool
+    _preserve_su_env: bool
+    _su_shell: str
+    _doas: bool
+    _doas_user: str
+
+    # Shell arguments
+    _shell_executable: str
+    _chdir: str
+    _env: Mapping[str, str]
+
+    # Connector control (outside of command generation)
+    _success_exit_codes: Iterable[int]
+    _timeout: int
+    _get_pty: bool
+    _stdin: Union[str, Iterable[str]]
+
+
+def generate_env(config: "Config", value: dict) -> dict:
+    env = config.ENV.copy()
+    env.update(value)
+    return env
+
+
+auth_argument_meta: dict[str, ArgumentMeta] = {
+    "_sudo": ArgumentMeta(
+        "Execute/apply any changes with sudo.",
+        default=lambda config: config.SUDO,
+    ),
+    "_sudo_user": ArgumentMeta(
+        "Execute/apply any changes with sudo as a non-root user.",
+        default=lambda config: config.SUDO_USER,
+    ),
+    "_use_sudo_login": ArgumentMeta(
+        "Execute sudo with a login shell.",
+        default=lambda config: config.USE_SUDO_LOGIN,
+    ),
+    "_sudo_password": ArgumentMeta(
+        "Password to sudo with. If needed and not specified pyinfra will prompt for it.",
+        default=lambda config: config.SUDO_PASSWORD,
+    ),
+    "_preserve_sudo_env": ArgumentMeta(
+        "Preserve the shell environment of the connecting user when using sudo.",
+        default=lambda config: config.PRESERVE_SUDO_ENV,
+    ),
+    "_su_user": ArgumentMeta(
+        "Execute/apply any changes with this user using su.",
+        default=lambda config: config.SU_USER,
+    ),
+    "_use_su_login": ArgumentMeta(
+        "Execute su with a login shell.",
+        default=lambda config: config.USE_SU_LOGIN,
+    ),
+    "_preserve_su_env": ArgumentMeta(
+        "Preserve the shell environment of the connecting user when using su.",
+        default=lambda config: config.PRESERVE_SU_ENV,
+    ),
+    "_su_shell": ArgumentMeta(
+        "Use this shell (instead of user login shell) when using ``_su``). "
+        + "Only available under Linux, for use when using `su` with a user that "
+        + "has nologin/similar as their login shell.",
+        default=lambda config: config.SU_SHELL,
+    ),
+    "_doas": ArgumentMeta(
+        "Execute/apply any changes with doas.",
+        default=lambda config: config.DOAS,
+    ),
+    "_doas_user": ArgumentMeta(
+        "Execute/apply any changes with doas as a non-root user.",
+        default=lambda config: config.DOAS_USER,
+    ),
+}
+
+shell_argument_meta: dict[str, ArgumentMeta] = {
+    "_shell_executable": ArgumentMeta(
+        "The shell executable to use for executing commands.",
+        default=lambda config: config.SHELL,
+    ),
+    "_chdir": ArgumentMeta(
+        "Directory to switch to before executing the command.",
+        default=lambda _: "",
+    ),
+    "_env": ArgumentMeta(
+        "Dictionary of environment variables to set.",
+        default=lambda _: {},
+        handler=generate_env,
+    ),
+    "_success_exit_codes": ArgumentMeta(
+        "List of exit codes to consider a success.",
+        default=lambda _: [0],
+    ),
+    "_timeout": ArgumentMeta(
+        "Timeout for *each* command executed during the operation.",
+        default=lambda _: None,
+    ),
+    "_get_pty": ArgumentMeta(
+        "Whether to get a pseudoTTY when executing any commands.",
+        default=lambda _: False,
+    ),
+    "_stdin": ArgumentMeta(
+        "String or buffer to send to the stdin of any commands.",
+        default=lambda _: None,
+    ),
+}
+
+
+# Meta arguments
+# These provide/extend additional operation metadata
+
+
+class MetaArguments(TypedDict):
+    name: str
+    _ignore_errors: bool
+    _continue_on_error: bool
+    _if: List[Callable[[], bool]]
+
+
+meta_argument_meta: dict[str, ArgumentMeta] = {
+    # NOTE: name is the only non-_-prefixed argument
+    "name": ArgumentMeta(
+        "Name of the operation.",
+        default=lambda _: None,
+    ),
+    "_ignore_errors": ArgumentMeta(
+        "Ignore errors when executing the operation.",
+        default=lambda config: config.IGNORE_ERRORS,
+    ),
+    "_continue_on_error": ArgumentMeta(
+        (
+            "Continue executing operation commands after error. "
+            "Only applies when ``_ignore_errors`` is true."
+        ),
+        default=lambda _: False,
+    ),
+    "_if": ArgumentMeta(
+        "Only run this operation if these functions returns True",
+        default=lambda _: [],
+    ),
+}
+
+
+# Execution arguments
+# These alter how pyinfra is to execute an operation. Notably these must all have the same value
+# over every target host for the same operation.
+
+
+class ExecutionArguments(TypedDict):
+    _parallel: int
+    _run_once: bool
+    _serial: bool
+
+
+execution_argument_meta: dict[str, ArgumentMeta] = {
+    "_parallel": ArgumentMeta(
+        "Run this operation in batches of hosts.",
+        default=lambda config: config.PARALLEL,
+    ),
+    "_run_once": ArgumentMeta(
+        "Only execute this operation once, on the first host to see it.",
+        default=lambda _: False,
+    ),
+    "_serial": ArgumentMeta(
+        "Run this operation host by host, rather than in parallel.",
+        default=lambda _: False,
+    ),
+}
+
+
+class AllArguments(ConnectorArguments, MetaArguments, ExecutionArguments):
+    pass
+
+
+all_argument_meta: dict[str, ArgumentMeta] = {
+    **auth_argument_meta,
+    **shell_argument_meta,
+    **meta_argument_meta,
+    **execution_argument_meta,
+}
+
+EXECUTION_KWARG_KEYS = list(ExecutionArguments.__annotations__.keys())
+CONNECTOR_ARGUMENT_KEYS = list(ConnectorArguments.__annotations__.keys())
+
+__argument_docs__ = {
+    "Privilege & user escalation": (
+        auth_argument_meta,
+        """
+        .. code:: python
+
+            # Execute a command with sudo
+            server.user(
+                name="Create pyinfra user using sudo",
+                user="pyinfra",
+                _sudo=True,
+            )
+
+            # Execute a command with a specific sudo password
+            server.user(
+                name="Create pyinfra user using sudo",
+                user="pyinfra",
+                _sudo=True,
+                _sudo_password="my-secret-password",
+            )
+        """,
+    ),
+    "Shell control & features": (
+        shell_argument_meta,
+        """
+        .. code:: python
+
+            # Execute from a specific directory
+            server.shell(
+                name="Bootstrap nginx params",
+                commands=["openssl dhparam -out dhparam.pem 4096"],
+                _chdir="/etc/ssl/certs",
+            )
+        """,
+    ),
+    "Operation meta & callbacks": (meta_argument_meta, ""),
+    "Execution strategy": (execution_argument_meta, ""),
+}
+
+
+def pop_global_arguments(
+    kwargs: dict[str, Any],
+    state: Optional["State"] = None,
+    host: Optional["Host"] = None,
+    keys_to_check=None,
+) -> tuple[AllArguments, list[str]]:
+    """
+    Pop and return operation global keyword arguments, in preferred order:
+
+    + From the current context (a direct @operator or @deploy function being called)
+    + From any current @deploy context (deploy kwargs)
+    + From the host data variables
+    + From the config variables
+    """
+
+    state = state or context.state
+    host = host or context.host
+
+    config = state.config
+    if context.ctx_config.isset():
+        config = context.config
+
+    meta_kwargs: dict[str, Any] = host.current_deploy_kwargs or {}  # type: ignore[assignment]
+
+    arguments: dict[str, Any] = {}
+    found_keys: list[str] = []
+
+    for key, type_ in get_type_hints(AllArguments).items():
+        if keys_to_check and key not in keys_to_check:
+            continue
+
+        argument_meta = all_argument_meta[key]
+        handler = argument_meta.handler
+        default: Any = argument_meta.default(config)
+
+        host_default = getattr(host.data, key, default_sentinel)
+        if host_default is not default_sentinel:
+            default = host_default
+
+        if key in kwargs:
+            found_keys.append(key)
+            value = kwargs.pop(key)
+        else:
+            value = meta_kwargs.get(key, default)
+
+        if handler:
+            value = handler(config, value)
+
+        if value != default:
+            raise_if_bad_type(
+                value,
+                type_,
+                ArgumentTypeError,
+                f"Invalid argument `{key}`:",
+            )
+
+        # TODO: why is type failing here?
+        arguments[key] = value  # type: ignore
+    return cast(AllArguments, arguments), found_keys

pyinfra-3.0b1/pyinfra/api/arguments_typed.py

@@ -0,0 +1,77 @@
+from __future__ import annotations
+
+from typing import (
+    TYPE_CHECKING,
+    Callable,
+    Generator,
+    Generic,
+    Iterable,
+    List,
+    Mapping,
+    Optional,
+    Union,
+)
+
+from typing_extensions import ParamSpec, Protocol
+
+if TYPE_CHECKING:
+    from pyinfra.api.operation import OperationMeta
+
+P = ParamSpec("P")
+
+
+# Unfortunately we have to re-type out all of the global arguments here because
+# Python typing doesn't (yet) support merging kwargs. This acts as the operation
+# decorator output function which merges actual operation args/kwargs in paramspec
+# with the global arguments available in all operations.
+# The nature of "global arguments" is somewhat opposed to static typing as it's
+# indirect and somewhat magic, but we are where we are.
+class PyinfraOperation(Generic[P], Protocol):
+    _inner: Callable[P, Generator]
+
+    def __call__(
+        self,
+        #
+        # ConnectorArguments
+        #
+        # Auth
+        _sudo: bool = False,
+        _sudo_user: Optional[str] = None,
+        _use_sudo_login: bool = False,
+        _sudo_password: Optional[str] = None,
+        _preserve_sudo_env: bool = False,
+        _su_user: Optional[str] = None,
+        _use_su_login: bool = False,
+        _preserve_su_env: bool = False,
+        _su_shell: Optional[str] = None,
+        _doas: bool = False,
+        _doas_user: Optional[str] = None,
+        # Shell arguments
+        _shell_executable: Optional[str] = None,
+        _chdir: Optional[str] = None,
+        _env: Optional[Mapping[str, str]] = None,
+        # Connector control
+        _success_exit_codes: Iterable[int] = (0,),
+        _timeout: Optional[int] = None,
+        _get_pty: bool = False,
+        _stdin: Union[None, str, list[str], tuple[str, ...]] = None,
+        #
+        # MetaArguments
+        #
+        name: Optional[str] = None,
+        _ignore_errors: bool = False,
+        _continue_on_error: bool = False,
+        _if: Optional[List[Callable[[], bool]]] = None,
+        #
+        # ExecutionArguments
+        #
+        _parallel: Optional[int] = None,
+        _run_once: bool = False,
+        _serial: bool = False,
+        #
+        # The op itself
+        #
+        *args: P.args,
+        **kwargs: P.kwargs,
+    ) -> "OperationMeta":
+        ...