pyinfra 3.5.1__tar.gz → 3.6.1__tar.gz

pyinfra-3.6.1/.claude/settings.local.json

@@ -0,0 +1,16 @@
+{
+  "permissions": {
+    "allow": [
+      "mcp__github__pull_request_read",
+      "Bash(python -m pytest:*)",
+      "Bash(uv run pytest:*)",
+      "Bash(python scripts/lint_arguments_sync.py:*)",
+      "mcp__github__issue_read",
+      "mcp__github__list_issues",
+      "Bash(grep:*)",
+      "mcp__github__list_tags",
+      "mcp__github__list_commits",
+      "mcp__github__search_users"
+    ]
+  }
+}

pyinfra-3.6.1/.claude/skills/generatechangelog/SKILL.md

@@ -0,0 +1,10 @@
+---
+name: generatechangelog
+---
+
+Generate pyinfra changelog for release:
+
+- get previous v3.x tag
+- use github mcp to pull commit messages, and github usernames
+- generate changelog in a similar format to CHANGELOG.md (grab the first ~100 lines to get an idea)
+- ask human overlord to confirm then prepend it to CHANGELOG.md

pyinfra-3.6.1/.github/workflows/test.yml

@@ -0,0 +1,125 @@
+name: Lint & Test
+
+on:
+  push:
+    branches:
+      - 3.x
+      - 2.x
+      - 1.x
+      - 0.x
+  pull_request:
+
+jobs:
+  # Linting
+  #
+
+  lint:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v5
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - name: Install uv
+        uses: astral-sh/setup-uv@v6
+      - uses: astral-sh/ruff-action@v3
+      - uses: astral-sh/ruff-action@v3
+        with:
+          args: "format --check"
+
+  type-check:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v5
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - name: Install uv
+        uses: astral-sh/setup-uv@v6
+      - run: uv sync --group test --no-default-groups
+      - run: uv run mypy
+      - run: uv run python scripts/lint_arguments_sync.py
+
+  spell-check:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v5
+      - name: Check spelling
+        uses: crate-ci/typos@v1.30.3
+
+  unit-test:
+    needs:
+      - lint
+      - type-check
+      - spell-check
+    strategy:
+      matrix:
+        os: [macos-26, macos-15, macos-14, windows-2025, windows-2022, ubuntu-24.04, ubuntu-22.04]
+        # Test every OS vs. Python 3.13, and only one for 3.1[012]
+        python-version: ["3.13"]
+        include:
+          - os: ubuntu-24.04
+            python-version: "3.12"
+          - os: ubuntu-24.04
+            python-version: "3.11"
+          - os: ubuntu-24.04
+            python-version: "3.10"
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v5
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install uv
+        uses: astral-sh/setup-uv@v6
+      - run: uv sync --group test --no-default-groups --python=${{ matrix.python-version }}
+      - run: uv run pytest --cov --disable-warnings
+      - name: Upload coverage report to codecov
+        if: ${{ matrix.os == 'ubuntu-24.04' && matrix.python-version == '3.13' }}
+        uses: codecov/codecov-action@v5
+
+  # End-to-end tests
+  #
+
+  end-to-end-test:
+    needs:
+      - lint
+      - type-check
+      - spell-check
+    strategy:
+      matrix:
+        os: [ubuntu-22.04, ubuntu-24.04]
+        test-name: [local, ssh, docker, podman]
+        include:
+          - os: macos-14
+            test-name: local
+          - os: macos-15
+            test-name: local
+          - os: macos-26
+            test-name: local
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v5
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - name: Install uv
+        uses: astral-sh/setup-uv@v6
+      - run: uv sync --group test --no-default-groups --python=${{ matrix.python-version }}
+      - run: uv run pytest -m end_to_end_${{ matrix.test-name }}
+
+  # Finally, single jon to check if all completed, for use in protected branch
+
+  tests-done:
+    if: ${{ always() }}
+    needs:
+      - lint
+      - type-check
+      - spell-check
+      - unit-test
+      - end-to-end-test
+    runs-on: ubuntu-latest
+    steps:
+      - uses: matrix-org/done-action@v3
+        with:
+          needs: ${{ toJSON(needs) }}

pyinfra-3.6.1/.issues/1019.md

@@ -0,0 +1,199 @@
+# GitHub Issue #1019: files.line grep inserts wrong regex
+
+## Issue Summary
+
+**Reported by:** JustScreaMy (October 4, 2023)
+**Status:** Open (bug label)
+**Assignee:** Fizzadar
+
+### Problem Description
+
+When using `files.line` to replace a single line, the operation produces an incorrect grep regex pattern, causing the line to never be matched and therefore automatically inserted at the end of the file instead of replacing the existing line.
+
+### Reproduction Code
+
+```python
+from pyinfra.operations import files
+
+files.line(
+    name="Disable password login",
+    path="/etc/ssh/sshd_config",
+    line="PasswordAuthentication .+",
+    replace="PasswordAuthentication no"
+)
+```
+
+### Expected Behavior
+
+The code should match a line containing `PasswordAuthentication yes` and replace it with `PasswordAuthentication no`.
+
+### Actual Behavior (at time of report)
+
+The grep command generated was:
+```bash
+sh -c 'grep -e '\"'\"'^.*.+PasswordAuthentication.+.*$'\"'\"' /etc/ssh/sshd_config
+```
+
+The problem was that the original regex `PasswordAuthentication .+` was being transformed into `^.*.+PasswordAuthentication.+.*$` which places the `.+` from the original pattern **before** the literal text instead of after it.
+
+---
+
+## Root Cause Analysis
+
+### Investigation Findings
+
+Based on the comments on the issue and analysis of the current codebase:
+
+1. **Issue Comment from "bad" (June 19, 2025):** States that the `.+` insertion issue appears to be fixed on 3.x HEAD, with the example grep now showing:
+   ```
+   sh -c 'grep -e '\"'\"'^.*10.0.2.2[        ]\\{1,\\}host.*$'\"'\"' /etc/hosts [...]
+   ```
+
+2. **Current Code Path:**
+
+   The `files.line` operation (in `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/files.py`, line 278-475) calls `adjust_regex()` (line 372) to transform the user-provided line pattern.
+
+   The `adjust_regex()` function (in `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/util/files.py`, lines 200-216):
+   ```python
+   def adjust_regex(line: str, escape_regex_characters: bool) -> str:
+       """
+       Ensure the regex starts with '^' and ends with '$' and escape regex characters if requested
+       """
+       match_line = line
+
+       if escape_regex_characters:
+           match_line = re.sub(r"([\.\\\+\*\?\[\^\]\$\(\)\{\}\-])", r"\\\1", match_line)
+
+       # Ensure we're matching a whole line, note: match may be a partial line so we
+       # put any matches on either side.
+       if not match_line.startswith("^"):
+           match_line = "^.*{0}".format(match_line)
+       if not match_line.endswith("$"):
+           match_line = "{0}.*$".format(match_line)
+
+       return match_line
+   ```
+
+3. **Historical Bug Context:**
+
+   Based on the original report's grep output `^.*.+PasswordAuthentication.+.*$`, it appears the old code was incorrectly inserting `^.*.+` at the beginning regardless of the pattern contents. This has been fixed in the current implementation which simply prepends `^.*` (without `.+`).
+
+### Current Behavior Analysis
+
+For the input `line="PasswordAuthentication .+"`:
+- Current code produces: `^.*PasswordAuthentication .+.*$`
+- This regex will match any line containing "PasswordAuthentication " followed by one or more characters
+
+The current implementation **appears correct** for the reported issue.
+
+### Remaining Concerns
+
+As noted in the second comment by "bad", there are still potential optimization issues:
+
+1. **Unnecessary `^.*` prefix:** Adding `^.*` serves no purpose in grep matching and causes excessive backtracking, slowing down pattern matching on large files.
+
+2. **Unnecessary `.*$` suffix:** This is also redundant since grep already matches partial lines by default.
+
+These are performance concerns rather than correctness issues.
+
+---
+
+## Current Status Assessment
+
+Based on the investigation, **the original bug appears to have been fixed** in the current 3.x codebase. The `adjust_regex()` function now correctly prepends `^.*` (without the erroneous `.+`) and appends `.*$`.
+
+However, there are two perspectives on whether the issue should be closed:
+
+### Option 1: Close as Fixed
+The original reported behavior (`.+` being incorrectly inserted) is no longer present.
+
+### Option 2: Address Performance Concerns
+While functionally correct, the regex transformation adds unnecessary overhead:
+- `^.*` causes backtracking in grep
+- `.*$` is redundant
+
+---
+
+## Proposed Fix (Performance Optimization)
+
+If the decision is to address the performance concerns raised in the comments, the `adjust_regex()` function could be simplified:
+
+### Current Code (`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/util/files.py`):
+```python
+def adjust_regex(line: str, escape_regex_characters: bool) -> str:
+    """
+    Ensure the regex starts with '^' and ends with '$' and escape regex characters if requested
+    """
+    match_line = line
+
+    if escape_regex_characters:
+        match_line = re.sub(r"([\.\\\+\*\?\[\^\]\$\(\)\{\}\-])", r"\\\1", match_line)
+
+    # Ensure we're matching a whole line, note: match may be a partial line so we
+    # put any matches on either side.
+    if not match_line.startswith("^"):
+        match_line = "^.*{0}".format(match_line)
+    if not match_line.endswith("$"):
+        match_line = "{0}.*$".format(match_line)
+
+    return match_line
+```
+
+### Proposed Optimized Code:
+```python
+def adjust_regex(line: str, escape_regex_characters: bool) -> str:
+    """
+    Ensure the regex matches a whole line and escape regex characters if requested.
+
+    The pattern is wrapped to ensure it matches complete lines, not partial matches
+    within a line.
+    """
+    match_line = line
+
+    if escape_regex_characters:
+        match_line = re.sub(r"([\.\\\+\*\?\[\^\]\$\(\)\{\}\-])", r"\\\1", match_line)
+
+    # Ensure we're matching a whole line by anchoring at start and end.
+    # Note: We only add anchors, not wildcards, since grep already matches
+    # partial lines and adding ^.* causes unnecessary backtracking.
+    if not match_line.startswith("^"):
+        match_line = "^{0}".format(match_line)
+    if not match_line.endswith("$"):
+        match_line = "{0}$".format(match_line)
+
+    return match_line
+```
+
+**IMPORTANT:** This change would alter the matching behavior:
+- Current: `PasswordAuthentication .+` becomes `^.*PasswordAuthentication .+.*$` (matches line containing the pattern anywhere)
+- Proposed: `PasswordAuthentication .+` becomes `^PasswordAuthentication .+$` (matches line starting with the pattern)
+
+This is a **breaking change** and may not be desirable. The docstring for `files.line` states:
+> "Unless line matches a line (starts with ^, ends $), pyinfra will wrap it such that it does, like: `^.*LINE.*$`. This means we don't swap parts of lines out."
+
+So the current behavior is **intentional** to support matching partial lines. The performance cost is a documented trade-off for flexibility.
+
+---
+
+## Related Files
+
+1. **Operation definition:** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/files.py` (lines 278-475)
+2. **Regex adjustment utility:** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/util/files.py` (lines 200-216)
+3. **FindInFile fact (grep execution):** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/facts/files.py` (lines 423-456)
+
+---
+
+## Recommendations
+
+1. **Confirm the original bug is fixed:** The core issue (`.+` being inserted incorrectly) appears resolved.
+
+2. **Decision needed on performance optimization:**
+   - If keeping current behavior (partial line matching with `^.*...*$`), document that this is intentional
+   - If changing to strict anchoring (`^...$`), this is a breaking change requiring a major version bump or deprecation period
+
+3. **Consider closing the issue** with a note that:
+   - The original regex insertion bug is fixed
+   - The `^.*` and `.*$` wrapping is intentional per the documented behavior
+   - Users who want exact line matching should use explicit `^` and `$` anchors in their patterns
+
+4. **Update documentation** to clarify the performance implications of the default partial-line matching behavior and recommend explicit anchors for performance-critical use cases.

pyinfra-3.6.1/.issues/1026.md

@@ -0,0 +1,299 @@
+# GitHub Issue #1026: operations.apt.packages shows "SUCCESS/Changed" for packages that are already installed in some cases
+
+**Issue URL:** https://github.com/pyinfra-dev/pyinfra/issues/1026
+**Status:** Open
+**Reporter:** @stone-w4tch3r
+**Labels:** bug
+**Created:** November 6, 2023
+
+## Issue Summary
+
+When using `apt.packages` with package names that are "aliases" (i.e., virtual packages or packages that resolve to different actual package names), pyinfra incorrectly reports "SUCCESS/Changed" on every run, even when the package is already installed.
+
+### Reproduction Steps
+
+1. Run Ubuntu/Debian VM
+2. Run pyinfra with:
+```python
+from pyinfra.operations import apt
+
+apt.update(_sudo=True, cache_time=86400)
+apt.packages(_sudo=True, packages=['fortune'])
+```
+3. Run again - pyinfra reports "Changed: 1" even though the package was already installed
+
+### The Problem
+
+APT has a concept of package aliases/virtual packages. When you request `fortune`, APT resolves it to `fortune-mod`:
+
+```
+sudo apt install fortune -y
+Reading package lists... Done
+Building dependency tree
+Reading state information... Done
+Note, selecting 'fortune-mod' instead of 'fortune'  # <-- THIS
+fortune-mod is already the newest version (1:1.99.1-7build1).
+```
+
+pyinfra's `DebPackages` fact uses `dpkg -l` to list installed packages, which only shows the real package name (`fortune-mod`), not the alias (`fortune`). When the operation checks if `fortune` is installed, it doesn't find it because only `fortune-mod` is in the installed packages list.
+
+## Root Cause Analysis
+
+### Current Code Flow
+
+1. **`apt.packages` operation** (`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/apt.py`, lines 398-496):
+   - Calls `ensure_packages()` with the list of packages and current installed packages from `DebPackages` fact
+
+2. **`DebPackages` fact** (`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/facts/deb.py`, lines 30-58):
+   - Uses `dpkg -l` command to get installed packages
+   - Returns dict of `{package_name: [versions]}`
+   - Only knows about **actual** package names, not aliases
+
+3. **`ensure_packages` utility** (`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/util/packaging.py`, lines 136-231):
+   - Takes an `expand_package_fact` parameter that can resolve package names to their actual counterparts
+   - Uses `_has_package()` function to check if a package is installed
+   - **APT operations do NOT use `expand_package_fact`** - this is the root cause
+
+### How Pacman Solved This
+
+The pacman package manager has the same problem with groups and virtual packages. pyinfra solves it with:
+
+1. **`PacmanUnpackGroup` fact** (`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/facts/pacman.py`, lines 14-39):
+   ```python
+   class PacmanUnpackGroup(FactBase):
+       def command(self, package):
+           return 'pacman -S --print-format "%n" {0} || true'.format(shlex.quote(package))
+   ```
+   This resolves package aliases/groups to their actual package names.
+
+2. **pacman.packages operation** (`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/pacman.py`, lines 74-82):
+   ```python
+   yield from ensure_packages(
+       host,
+       packages,
+       host.get_fact(PacmanPackages),
+       present,
+       install_command="pacman --noconfirm -S",
+       uninstall_command="pacman --noconfirm -R",
+       expand_package_fact=lambda package: host.get_fact(PacmanUnpackGroup, package=package),
+   )
+   ```
+
+### The Missing Piece for APT
+
+The `apt.packages` operation does NOT pass an `expand_package_fact` parameter:
+
+```python
+# Current code in apt.py (line 486-496)
+yield from ensure_packages(
+    host,
+    packages,
+    host.get_fact(DebPackages),
+    present,
+    install_command=noninteractive_apt(install_command, force=force),
+    uninstall_command=noninteractive_apt(uninstall_command, force=force),
+    upgrade_command=noninteractive_apt(upgrade_command, force=force),
+    version_join="=",
+    latest=latest,
+)
+```
+
+## Proposed Fix
+
+### 1. Create a new `DebResolvePackage` fact
+
+Add to `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/facts/deb.py`:
+
+```python
+class DebResolvePackage(FactBase):
+    """
+    Resolves a package name (which may be an alias/virtual package) to the actual
+    package name(s) that would be installed by apt.
+
+    Returns a list of package names, or the original package if resolution fails.
+    """
+
+    @override
+    def requires_command(self, package) -> str:
+        return "apt-cache"
+
+    default = list
+
+    @override
+    def command(self, package):
+        # apt-cache show will show the resolved package info
+        # Use apt-get --simulate install to see what would actually be installed
+        # The -qq flag minimizes output, and we parse the "Inst" lines
+        return (
+            "apt-get -qq --simulate install {0} 2>/dev/null | "
+            "grep -oP '^Inst \\K[^ ]+' || echo {0}"
+        ).format(shlex.quote(package))
+
+    @override
+    def process(self, output):
+        # Return list of package names that would be installed
+        # Filter out empty lines
+        packages = [line.strip() for line in output if line.strip()]
+        return packages if packages else []
+```
+
+**Alternative approach using `apt-cache`:**
+
+```python
+class DebResolvePackage(FactBase):
+    """
+    Resolves a package name (which may be an alias/virtual package) to the actual
+    package name(s) that would be installed by apt.
+    """
+
+    @override
+    def requires_command(self, package) -> str:
+        return "apt-cache"
+
+    default = list
+
+    @override
+    def command(self, package):
+        # apt-cache policy shows the resolved package name
+        # For virtual packages, it shows what provides them
+        return "apt-cache policy {0} 2>/dev/null | head -1 | cut -d: -f1".format(
+            shlex.quote(package)
+        )
+
+    @override
+    def process(self, output):
+        result = []
+        for line in output:
+            line = line.strip()
+            if line:
+                result.append(line)
+        return result
+```
+
+**Recommended approach using `apt-cache showpkg`** (as suggested in GitHub comment):
+
+```python
+class DebResolvePackage(FactBase):
+    """
+    Resolves a package name (which may be an alias/virtual package) to the actual
+    package name(s) that would be installed by apt.
+
+    For virtual packages like 'fortune', this returns the real package name(s)
+    that provide the virtual package (e.g., 'fortune-mod').
+    """
+
+    @override
+    def requires_command(self, package) -> str:
+        return "apt-cache"
+
+    default = list
+
+    @override
+    def command(self, package):
+        # apt-cache showpkg shows reverse provides which tells us what packages
+        # provide a virtual package. We check if it's a real package first,
+        # and if not, look for providers.
+        return (
+            "if apt-cache show {0} >/dev/null 2>&1; then "
+            "echo {0}; "
+            "else "
+            "apt-cache showpkg {0} 2>/dev/null | "
+            "awk '/^Reverse Provides:/{{flag=1; next}} flag && NF{{print $1}}'; "
+            "fi"
+        ).format(shlex.quote(package))
+
+    @override
+    def process(self, output):
+        packages = [line.strip() for line in output if line.strip()]
+        return packages if packages else []
+```
+
+### 2. Update apt.packages to use the new fact
+
+Modify `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/apt.py`:
+
+```python
+# Add import at top
+from pyinfra.facts.deb import DebPackage, DebPackages, DebResolvePackage
+
+# Update ensure_packages call (around line 486)
+yield from ensure_packages(
+    host,
+    packages,
+    host.get_fact(DebPackages),
+    present,
+    install_command=noninteractive_apt(install_command, force=force),
+    uninstall_command=noninteractive_apt(uninstall_command, force=force),
+    upgrade_command=noninteractive_apt(upgrade_command, force=force),
+    version_join="=",
+    latest=latest,
+    expand_package_fact=lambda package: host.get_fact(DebResolvePackage, package=package),
+)
+```
+
+### 3. Add tests
+
+Create test cases in `/Users/nick/Dev/Webroot/pyinfra/src/tests/`:
+
+```python
+# Test for the new fact
+def test_deb_resolve_package_virtual():
+    """Test that virtual packages are resolved to their providers."""
+    # Test with a known virtual package like 'fortune' -> 'fortune-mod'
+    pass
+
+def test_deb_resolve_package_real():
+    """Test that real packages resolve to themselves."""
+    pass
+
+# Test for the operation
+def test_apt_packages_virtual_package_already_installed():
+    """Test that apt.packages doesn't report changes for already-installed virtual packages."""
+    pass
+```
+
+## Implementation Notes
+
+### Choice of apt command
+
+Several options were discussed in the GitHub comments:
+
+1. **`apt-cache showpkg`** - Shows package information including "Reverse Provides" which lists what packages provide a virtual package
+2. **`apt-file`** - Can find which package provides a file (overkill for this use case)
+3. **`apt-get --simulate install`** - Shows exactly what would be installed
+
+The recommended approach is `apt-cache showpkg` because:
+- It's available on all Debian/Ubuntu systems with apt
+- It directly shows the "Reverse Provides" relationship
+- It doesn't require simulating an install (faster)
+
+### Edge Cases to Consider
+
+1. **Multiple providers**: A virtual package might have multiple providers. The current implementation would need to check if ANY of them is installed.
+2. **Version-specific aliases**: Some package aliases might be version-specific.
+3. **Architecture-specific packages**: Packages like `libc6:amd64` need proper handling.
+4. **Transitional packages**: Packages that exist but recommend/depend on their successor.
+
+### Performance Considerations
+
+Using `expand_package_fact` will result in additional fact-gathering for each package. This is the same approach used by pacman and is acceptable. The facts are memoized, so repeated calls for the same package don't result in extra commands.
+
+## Related Files
+
+| File Path | Purpose |
+|-----------|---------|
+| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/apt.py` | APT operations including `apt.packages` |
+| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/facts/deb.py` | DEB package facts - needs new `DebResolvePackage` fact |
+| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/util/packaging.py` | `ensure_packages` utility that supports `expand_package_fact` |
+| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/pacman.py` | Reference implementation using `expand_package_fact` |
+| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/facts/pacman.py` | Reference `PacmanUnpackGroup` fact |
+
+## Conclusion
+
+The issue occurs because pyinfra's APT package management doesn't resolve package aliases/virtual packages to their actual names before checking if they're installed. The fix requires:
+
+1. Creating a new `DebResolvePackage` fact that uses `apt-cache showpkg` to resolve package names
+2. Updating `apt.packages` to pass the new fact as `expand_package_fact` to `ensure_packages`
+3. Adding appropriate tests
+
+This mirrors the existing solution for pacman packages and leverages the already-implemented `expand_package_fact` parameter in `ensure_packages()`.