pyinfra 3.0b4__tar.gz → 3.0.1__tar.gz

pyinfra-3.0.1/CHANGELOG.md

@@ -0,0 +1,73 @@
+# v3.0.1
+
+- Switch to `command -v` not `which` in `server.Which` fact (@lemmi)
+- Fix detection of xbps in `server.packages` operation (@romain-dartigues)
+- Fix argument typo in operations doc (@scoufman)
+- Add expanded note about detected changes + hidden side effects during execution
+- Fix missing global arguments in group data files
+- Fix `--group-data` CLI argument behaviour
+- Remove unused/dead `--quiet` flag
+
+# v3.0
+
+Welcome to pyinfra v3! This version is the biggest overhaul of pyinfra since it was created back in 2015. Most v2 deployment code should be automatically compatible, but as always be aware. Major changes:
+
+### Runtime operation execution
+
+pyinfra now executes operations at runtime, rather than pre-generating commands. Although the change isn't noticeable this fixes an entire class of bugs and confusion. See the [limitations](https://docs.pyinfra.com/en/2.x/deploy-process.html#limitations) section in the v2 docs. All of those issues are now a thing of the past.
+
+This represents a huge overhaul of pyinfra's internals and should be a huge improvement for users.
+
+Care has been taken to reduce the overhead of this change which still supports the same diffs and change proposal mechanism.
+
+### CLI flow & prompts
+
+The pyinfra CLI will now prompt (instead of ignore, or immediately exit) when problems are encountered, allowing the user to choose to continue. Additionally an approval step is added before executing changes (this can be skipped with `-y` or setting the `PYINFRA_YES` environment variable).
+
+### Extendable connectors API, typing overhaul
+
+v3 of pyinfra includes for the first time a (mostly) typed internal API with proper support for IDE linting. There's a whole new connectors API that provides a framework for building new connectors.
+
+### Breaking changes
+
+- Rename `_use_sudo_password` argument to `_sudo_password`
+- Remove `winrm` connector and `windows*` operations/facts, moving to [`pyinfra-windows`](https://github.com/pyinfra-dev/pyinfra-windows)
+- The deploy decorator must now be called, ie used as `@deploy()`, and is now typed
+- Remove broken Ansible inventory connector
+
+### Operations & Facts
+
+- Add `docker.container`, `docker.image`, `docker.volume`, `docker.network` & `docker.prune` operations (@apecnascimento)
+- Add `runit.service` operation and `RunitStatus` fact (@lemmi)
+- Add `TmpDir` fact
+- Add `services` argument to systemd facts for filtering
+- Add type hints for all the operations (@stone-w4tch3r)
+- Lowercase pip packages properly (PEP-0426)
+- Rename `postgresql` -> `postgres` operations & facts (old ones still work)
+- Improve IP/MAC parsing in `NetworkDevices` fact (@sudoBash418)
+- Enable getting `Home` fact for other users (@matthijskooijman)
+- Use users correct home directory in `server.user_authorized_keys` operation (@matthijskooijman)
+- Fix `destination`/`not_destination` arguments in `iptables.rule` operation
+- Fix remote dirs when executing from Windows in `files.sync` operation (@Renerick)
+- Fix quoting of systemd unit names (@martenlienen)
+
+### Other Changes
+
+- Add new `_if` global argument to control operation execution at runtime
+- Add `--debug-all` flag to set debug logging for all packages
+- Retry SSH connections on failure (configurable, see [SSH connector](https://docs.pyinfra.com/en/3.x/connectors/ssh.html#available-data)) (@fwiesel)
+- Documentation typo fixes (@szepeviktor, @sudoBash418)
+- Fix handling of binary files in Docker connector (@matthijskooijman)
+- Add `will_change` attribute and `did_change` context manager to `OperationMeta`
+- Replace use of `pkg_resources` with `importlib.metadata` (@diazona)
+- Fix identifying Python inventory files as modules (@martenlienen)
+- Fix typed arguments order (@cdleonard)
+- Check that fact commands don't take global arguments (@martenlienen)
+
+# v2.x
+
+[See this file in the `2.x` branch](https://github.com/Fizzadar/pyinfra/blob/2.x/CHANGELOG.md).
+
+# v1.x
+
+[See this file in the `1.x` branch](https://github.com/Fizzadar/pyinfra/blob/1.x/CHANGELOG.md).

{pyinfra-3.0b4/pyinfra.egg-info → pyinfra-3.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pyinfra
-Version: 3.0b4
+Version: 3.0.1
 Summary: pyinfra automates/provisions/manages/deploys infrastructure.
 Home-page: https://pyinfra.com
 Author: Nick / Fizzadar
@@ -92,10 +92,6 @@ Requires-Dist: redbaron; extra == "dev"
     </a>
 </p>
 
-<p>
-    <strong>Note: this is the v3 branch, which is currently in beta. <a href="https://docs.pyinfra.com/en/next">See the docs for v3</a>. If needed the <a href="https://github.com/pyinfra-dev/pyinfra/tree/2.x/">2.x branch is here</a>, but is in bugfix only mode.</strong>
-</p>
-
 <p>
     pyinfra turns Python code into shell commands and runs them on your servers. Execute ad-hoc commands and write declarative operations. Target SSH servers, local machine and Docker containers. Fast and scales from one server to thousands. Think <code>ansible</code> but Python instead of YAML, and a lot faster.
 </p>

{pyinfra-3.0b4 → pyinfra-3.0.1}/README.md

@@ -4,10 +4,6 @@
     </a>
 </p>
 
-<p>
-    <strong>Note: this is the v3 branch, which is currently in beta. <a href="https://docs.pyinfra.com/en/next">See the docs for v3</a>. If needed the <a href="https://github.com/pyinfra-dev/pyinfra/tree/2.x/">2.x branch is here</a>, but is in bugfix only mode.</strong>
-</p>
-
 <p>
     pyinfra turns Python code into shell commands and runs them on your servers. Execute ad-hoc commands and write declarative operations. Target SSH servers, local machine and Docker containers. Fast and scales from one server to thousands. Think <code>ansible</code> but Python instead of YAML, and a lot faster.
 </p>

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/api/facts.py

@@ -14,18 +14,7 @@ import inspect
 import re
 from inspect import getcallargs
 from socket import error as socket_error, timeout as timeout_error
-from typing import (
-    TYPE_CHECKING,
-    Any,
-    Callable,
-    Generic,
-    Iterable,
-    Optional,
-    Type,
-    TypeVar,
-    Union,
-    cast,
-)
+from typing import TYPE_CHECKING, Any, Callable, Generic, Iterable, Optional, Type, TypeVar, cast
 
 import click
 import gevent
@@ -38,7 +27,6 @@ from pyinfra.api.util import (
     get_kwargs_str,
     log_error_or_warning,
     log_host_command_error,
-    make_hash,
     print_host_combined_output,
 )
 from pyinfra.connectors.util import CommandOutput
@@ -66,11 +54,12 @@ class FactBase(Generic[T]):
 
     abstract: bool = True
 
-    shell_executable: Optional[str] = None
+    shell_executable: str | None = None
 
-    requires_command: Optional[str] = None
+    command: Callable[..., str | StringCommand]
 
-    command: Union[str, Callable]
+    def requires_command(self, *args, **kwargs) -> str | None:
+        return None
 
     def __init_subclass__(cls) -> None:
         super().__init_subclass__()
@@ -113,8 +102,7 @@ class ShortFactBase(Generic[T]):
         module_name = cls.__module__.replace("pyinfra.facts.", "")
         cls.name = f"{module_name}.{cls.__name__}"
 
-    @staticmethod
-    def process_data(data):
+    def process_data(self, data):
         return data
 
 
@@ -130,30 +118,6 @@ def _make_command(command_attribute, host_args):
     return command_attribute
 
 
-def _get_executor_kwargs(
-    state: "State",
-    host: "Host",
-    override_kwargs: Optional[dict[str, Any]] = None,
-    override_kwarg_keys: Optional[list[str]] = None,
-):
-    if override_kwargs is None:
-        override_kwargs = {}
-    if override_kwarg_keys is None:
-        override_kwarg_keys = []
-
-    # Use the current operation global kwargs, or generate defaults
-    global_kwargs = host.current_op_global_arguments
-    if not global_kwargs:
-        global_kwargs, _ = pop_global_arguments({}, state, host)
-
-    # Apply any current op kwargs that *weren't* found in the overrides
-    override_kwargs.update(
-        {key: value for key, value in global_kwargs.items() if key not in override_kwarg_keys},
-    )
-
-    return {key: value for key, value in override_kwargs.items() if key in CONNECTOR_ARGUMENT_KEYS}
-
-
 def _handle_fact_kwargs(state, host, cls, args, kwargs):
     args = args or []
     kwargs = kwargs or {}
@@ -296,7 +260,7 @@ def _get_fact(
         log_host_command_error(
             host,
             e,
-            timeout=global_kwargs["_timeout"],
+            timeout=global_kwargs.get("_timeout"),
         )
 
     stdout_lines, stderr_lines = output.stdout_lines, output.stderr_lines
@@ -346,13 +310,6 @@ def _get_fact(
     return data
 
 
-def _get_fact_hash(state: "State", host: "Host", cls, args, kwargs):
-    if issubclass(cls, ShortFactBase):
-        cls = cls.fact
-    fact_kwargs, executor_kwargs = _handle_fact_kwargs(state, host, cls, args, kwargs)
-    return make_hash((cls, fact_kwargs, executor_kwargs))
-
-
 def get_host_fact(
     state: "State",
     host: "Host",

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/api/util.py

@@ -241,7 +241,7 @@ def log_error_or_warning(
     )
 
 
-def log_host_command_error(host: "Host", e: Exception, timeout: int = 0) -> None:
+def log_host_command_error(host: "Host", e: Exception, timeout: int | None = 0) -> None:
     if isinstance(e, timeout_error):
         logger.error(
             "{0}{1}".format(

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/connectors/ssh.py

@@ -123,7 +123,7 @@ class SSHConnector(BaseConnector):
 
         hosts = (
             ["my-host-1.net", "my-host-2.net"],
-            {"ssh_username": "ssh-user"},
+            {"ssh_user": "ssh-user"},
         )
 
     Multiple hosts with different SSH usernames:
@@ -131,8 +131,8 @@ class SSHConnector(BaseConnector):
     .. code:: python
 
         hosts = [
-            ("my-host-1.net", {"ssh_username": "ssh-user"}),
-            ("my-host-2.net", {"ssh_username": "other-user"}),
+            ("my-host-1.net", {"ssh_user": "ssh-user"}),
+            ("my-host-2.net", {"ssh_user": "other-user"}),
         ]
     """
 

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/connectors/sshuserclient/client.py

@@ -124,7 +124,7 @@ class SSHClient(ParamikoClient):
     original idea at http://bitprophet.org/blog/2012/11/05/gateway-solutions/.
     """
 
-    def connect(
+    def connect(  # type: ignore[override]
         self,
         hostname,
         _pyinfra_ssh_forward_agent=None,

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/facts/apk.py

@@ -18,8 +18,11 @@ class ApkPackages(FactBase):
         }
     """
 
-    command = "apk list --installed"
-    requires_command = "apk"
+    def command(self) -> str:
+        return "apk list --installed"
+
+    def requires_command(self) -> str:
+        return "apk"
 
     default = dict
 

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/facts/apt.py

@@ -52,11 +52,14 @@ class AptSources(FactBase):
         ]
     """
 
-    command = make_cat_files_command(
-        "/etc/apt/sources.list",
-        "/etc/apt/sources.list.d/*.list",
-    )
-    requires_command = "apt"  # if apt installed, above should exist
+    def command(self) -> str:
+        return make_cat_files_command(
+            "/etc/apt/sources.list",
+            "/etc/apt/sources.list.d/*.list",
+        )
+
+    def requires_command(self) -> str:
+        return "apt"  # if apt installed, above should exist
 
     default = list
 
@@ -86,5 +89,8 @@ class AptKeys(GpgFactBase):
     """
 
     # This requires both apt-key *and* apt-key itself requires gpg
-    command = "! command -v gpg || apt-key list --with-colons"
-    requires_command = "apt-key"
+    def command(self) -> str:
+        return "! command -v gpg || apt-key list --with-colons"
+
+    def requires_command(self) -> str:
+        return "apt-key"

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/facts/brew.py

@@ -37,18 +37,22 @@ class BrewVersion(FactBase):
 
     """
 
-    command = "brew --version"
-    requires_command = "brew"
+    def command(self) -> str:
+        return "brew --version"
+
+    def requires_command(self) -> str:
+        return "brew"
 
     @staticmethod
     def default():
         return [0, 0, 0]
 
     def process(self, output):
-        m = VERSION_MATCHER.match(output[0])
+        out = list(output)[0]
+        m = VERSION_MATCHER.match(out)
         if m is not None:
             return [int(m.group(key)) for key in ["major", "minor", "patch"]]
-        logger.warning("could not parse version string from brew: %s", output[0])
+        logger.warning("could not parse version string from brew: %s", out)
         return self.default()
 
 
@@ -63,8 +67,11 @@ class BrewPackages(FactBase):
         }
     """
 
-    command = "brew list --versions"
-    requires_command = "brew"
+    def command(self) -> str:
+        return "brew list --versions"
+
+    def requires_command(self) -> str:
+        return "brew"
 
     default = dict
 
@@ -83,11 +90,14 @@ class BrewCasks(BrewPackages):
         }
     """
 
-    command = (
-        r'if brew --version | grep -q -e "Homebrew\ +(1\.|2\.[0-5]).*" 1>/dev/null;'
-        r"then brew cask list --versions; else brew list --cask --versions; fi"
-    )
-    requires_command = "brew"
+    def command(self) -> str:
+        return (
+            r'if brew --version | grep -q -e "Homebrew\ +(1\.|2\.[0-5]).*" 1>/dev/null;'
+            r"then brew cask list --versions; else brew list --cask --versions; fi"
+        )
+
+    def requires_command(self) -> str:
+        return "brew"
 
 
 class BrewTaps(FactBase):
@@ -95,8 +105,11 @@ class BrewTaps(FactBase):
     Returns a list of brew taps.
     """
 
-    command = "brew tap"
-    requires_command = "brew"
+    def command(self) -> str:
+        return "brew tap"
+
+    def requires_command(self) -> str:
+        return "brew"
 
     default = list
 

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/facts/bsdinit.py

@@ -9,11 +9,12 @@ class RcdStatus(InitdStatus):
     BSD init scripts are well behaved and as such their output can be trusted.
     """
 
-    command = """
-        for SERVICE in `find /etc/rc.d /usr/local/etc/rc.d -type f`; do
-            $SERVICE status 2> /dev/null || $SERVICE check 2> /dev/null
-            echo "`basename $SERVICE`=$?"
-        done
-    """
+    def command(self) -> str:
+        return """
+            for SERVICE in `find /etc/rc.d /usr/local/etc/rc.d -type f`; do
+                $SERVICE status 2> /dev/null || $SERVICE check 2> /dev/null
+                echo "`basename $SERVICE`=$?"
+            done
+        """
 
     default = dict

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/facts/cargo.py

@@ -22,10 +22,11 @@ class CargoPackages(FactBase):
 
     default = dict
 
-    requires_command = "cargo"
-
-    def command(self):
+    def command(self) -> str:
         return "cargo install --list"
 
+    def requires_command(self) -> str:
+        return "cargo"
+
     def process(self, output):
         return parse_packages(CARGO_REGEX, output)

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/facts/choco.py

@@ -18,7 +18,9 @@ class ChocoPackages(FactBase):
         }
     """
 
-    command = "choco list"
+    def command(self) -> str:
+        return "choco list"
+
     shell_executable = "ps"
 
     default = dict
@@ -32,8 +34,8 @@ class ChocoVersion(FactBase):
     Returns the choco (Chocolatey) version.
     """
 
-    command = "choco --version"
+    def command(self) -> str:
+        return "choco --version"
 
-    @staticmethod
-    def process(output):
+    def process(self, output):
         return "".join(output).replace("\n", "")

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/facts/deb.py

@@ -16,8 +16,11 @@ class DebArch(FactBase):
     Returns the architecture string used in apt repository sources, eg ``amd64``.
     """
 
-    command = "dpkg --print-architecture"
-    requires_command = "dpkg"
+    def command(self) -> str:
+        return "dpkg --print-architecture"
+
+    def requires_command(self) -> str:
+        return "dpkg"
 
 
 class DebPackages(FactBase):
@@ -31,8 +34,11 @@ class DebPackages(FactBase):
         }
     """
 
-    command = "dpkg -l"
-    requires_command = "dpkg"
+    def command(self) -> str:
+        return "dpkg -l"
+
+    def requires_command(self) -> str:
+        return "dpkg"
 
     default = dict
 
@@ -55,7 +61,8 @@ class DebPackage(FactBase):
         "version": r"^Version:\s+({0})$".format(DEB_PACKAGE_VERSION_REGEX),
     }
 
-    requires_command = "dpkg"
+    def requires_command(self, package) -> str:
+        return "dpkg"
 
     def command(self, package):
         return "! test -e {0} && (dpkg -s {0} 2>/dev/null || true) || dpkg -I {0}".format(

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/facts/dnf.py

@@ -23,12 +23,15 @@ class DnfRepositories(FactBase):
         ]
     """
 
-    command = make_cat_files_command(
-        "/etc/dnf.conf",
-        "/etc/dnf.repos.d/*.repo",
-        "/etc/yum.repos.d/*.repo",
-    )
-    requires_command = "dnf"
+    def command(self) -> str:
+        return make_cat_files_command(
+            "/etc/dnf.conf",
+            "/etc/dnf.repos.d/*.repo",
+            "/etc/yum.repos.d/*.repo",
+        )
+
+    def requires_command(self) -> str:
+        return "dnf"
 
     default = list
 

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/facts/docker.py

@@ -9,7 +9,9 @@ class DockerFactBase(FactBase):
     abstract = True
 
     docker_type: str
-    requires_command = "docker"
+
+    def requires_command(self, *args, **kwargs) -> str:
+        return "docker"
 
     def process(self, output):
         output = "".join(output)
@@ -21,7 +23,8 @@ class DockerSystemInfo(DockerFactBase):
     Returns ``docker system info`` output in JSON format.
     """
 
-    command = 'docker system info --format="{{json .}}"'
+    def command(self) -> str:
+        return 'docker system info --format="{{json .}}"'
 
 
 # All Docker objects
@@ -33,7 +36,8 @@ class DockerContainers(DockerFactBase):
     Returns ``docker inspect`` output for all Docker containers.
     """
 
-    command = "docker container inspect `docker ps -qa`"
+    def command(self) -> str:
+        return "docker container inspect `docker ps -qa`"
 
 
 class DockerImages(DockerFactBase):
@@ -41,7 +45,8 @@ class DockerImages(DockerFactBase):
     Returns ``docker inspect`` output for all Docker images.
     """
 
-    command = "docker image inspect `docker images -q`"
+    def command(self) -> str:
+        return "docker image inspect `docker images -q`"
 
 
 class DockerNetworks(DockerFactBase):
@@ -49,7 +54,8 @@ class DockerNetworks(DockerFactBase):
     Returns ``docker inspect`` output for all Docker networks.
     """
 
-    command = "docker network inspect `docker network ls -q`"
+    def command(self) -> str:
+        return "docker network inspect `docker network ls -q`"
 
 
 # Single Docker objects
@@ -93,7 +99,8 @@ class DockerVolumes(DockerFactBase):
     Returns ``docker inspect`` output for all Docker volumes.
     """
 
-    command = "docker volume inspect `docker volume ls -q`"
+    def command(self) -> str:
+        return "docker volume inspect `docker volume ls -q`"
 
 
 class DockerVolume(DockerSingleMixin):

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/facts/files.py

@@ -312,8 +312,7 @@ class FindFilesBase(FactBase):
     default = list
     type_flag: str
 
-    @staticmethod
-    def process(output):
+    def process(self, output):
         return output
 
     def command(self, path, quote_path=True):
@@ -353,7 +352,8 @@ class Flags(FactBase):
     Returns a list of the file flags set for the specified file or directory.
     """
 
-    requires_command = "chflags"  # don't try to retrieve them if we can't set them
+    def requires_command(self, path) -> str:
+        return "chflags"  # don't try to retrieve them if we can't set them
 
     def command(self, path):
         return make_formatted_string_command(

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/facts/gem.py

@@ -18,8 +18,11 @@ class GemPackages(FactBase):
         }
     """
 
-    command = "gem list --local"
-    requires_command = "gem"
+    def command(self) -> str:
+        return "gem list --local"
+
+    def requires_command(self) -> str:
+        return "gem"
 
     default = dict
 

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/facts/git.py

@@ -5,32 +5,29 @@ import re
 from pyinfra.api.facts import FactBase
 
 
-class GitBranch(FactBase):
-    requires_command = "git"
+class GitFactBase(FactBase):
+    def requires_command(self, *args, **kwargs) -> str:
+        return "git"
 
-    @staticmethod
-    def command(repo):
+
+class GitBranch(GitFactBase):
+    def command(self, repo) -> str:
         return "! test -d {0} || (cd {0} && git describe --all)".format(repo)
 
-    @staticmethod
-    def process(output):
+    def process(self, output):
         return re.sub(r"(heads|tags)/", r"", "\n".join(output))
 
 
-class GitConfig(FactBase):
+class GitConfig(GitFactBase):
     default = dict
 
-    requires_command = "git"
-
-    @staticmethod
-    def command(repo=None):
+    def command(self, repo=None) -> str:
         if repo is None:
             return "git config --global -l || true"
 
         return "! test -d {0} || (cd {0} && git config --local -l)".format(repo)
 
-    @staticmethod
-    def process(output):
+    def process(self, output):
         items: dict[str, list[str]] = {}
 
         for line in output:
@@ -40,19 +37,15 @@ class GitConfig(FactBase):
         return items
 
 
-class GitTrackingBranch(FactBase):
-    requires_command = "git"
-
-    @staticmethod
-    def command(repo):
+class GitTrackingBranch(GitFactBase):
+    def command(self, repo) -> str:
         return r"! test -d {0} || (cd {0} && git status --branch --porcelain)".format(repo)
 
-    @staticmethod
-    def process(output):
+    def process(self, output):
         if not output:
             return None
 
-        m = re.search(r"\.{3}(\S+)\b", output[0])
+        m = re.search(r"\.{3}(\S+)\b", list(output)[0])
         if m:
             return m.group(1)
         return None

{pyinfra-3.0b4 → pyinfra-3.0.1}/pyinfra/facts/gpg.py

@@ -8,7 +8,8 @@ from pyinfra.api import FactBase
 class GpgFactBase(FactBase):
     abstract = True
 
-    requires_command = "gpg"
+    def requires_command(self, *args, **kwargs) -> str:
+        return "gpg"
 
     key_record_type = "pub"
     subkey_record_type = "sub"