pkgeter 1.1.0__tar.gz

pkgeter-1.1.0/PKG-INFO

@@ -0,0 +1,148 @@
+Metadata-Version: 2.4
+Name: pkgeter
+Version: 1.1.0
+Summary: Offline package downloader - support Debian/apt and RPM/dnf, resolve deps, download packages, generate offline install script
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/mlzxgzy/pkgeter
+Project-URL: Repository, https://github.com/mlzxgzy/pkgeter.git
+Project-URL: BugTracker, https://github.com/mlzxgzy/pkgeter/issues
+Keywords: debian,rpm,centos,offline,package,apt,dnf,yum,downloader
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: System Administrators
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: System :: Archiving :: Packaging
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: httpx>=0.25
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: pyreadline3; sys_platform == "win32"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+
+# pkgeter <small>v1.1</small>
+
+**English** | [中文](README_CH.md)
+
+**Offline package downloader** — supports **Debian/apt** and **RPM/dnf** (CentOS Stream). Resolve dependency trees, download `.deb` or `.rpm` files, and generate an offline install script.
+
+Works on any platform (Linux, Windows, macOS) — useful when you need to install packages on an air-gapped or offline machine.
+
+## Features
+
+- **Dual backend** — supports Debian (`dpkg`) and RPM (`rpm`) based distributions
+- **Distribution presets** — one-command selection: `--distro debian-bookworm`, `--distro centos-9`
+- **Interactive REPL** — run `pkgeter` with no arguments to enter a switch-style CLI with prefix matching and TAB completion
+- **Multi-repo merge** — automatically combines repositories (e.g., main + security, BaseOS + AppStream + EPEL)
+- **Dependency resolution** — recursively resolves all dependencies for the target packages
+- **Skip installed packages** — optionally provide a `dpkg -l` output to skip already-installed packages
+- **SHA256 verification** — validates every downloaded `.deb` or `.rpm` file
+- **Source caching** — caches repository metadata with SHA256 validation (like APT), only re-downloads when changed
+- **Offline install script** — auto-generates `install.sh` that runs `dpkg -i` or `rpm -ivh` in dependency order
+- **Multiple mirrors** — specify fallback mirrors, tried in order until one succeeds
+- **Persistent config** — preferences saved to `~/.config/pkgeter/config.yaml`
+- **Repo management** — add, list, and remove custom repositories via `pkgeter repo`
+
+## Installation
+### From source
+
+```bash
+git clone https://github.com/mlzxgzy/pkgeter.git
+cd pkgeter
+pip install -e .
+```
+
+## Usage
+
+```bash
+# Interactive REPL (no arguments)
+pkgeter
+
+# Download packages using a distribution preset
+pkgeter get -p nginx --distro debian-bookworm
+pkgeter get -p nginx --distro centos-9
+pkgeter get -p nginx --distro debian-bullseye
+
+# Short prefix forms work too
+pkgeter g -p nginx --distro centos-9
+
+# Legacy usage (backward compatible)
+pkgeter get -p vim
+pkgeter get -p nginx -r bookworm -a amd64
+
+# Specify multiple mirrors (tried in order)
+pkgeter get -p nginx -m https://deb.debian.org/debian -m https://ftp.debian.org/debian
+
+# Manage repositories
+pkgeter repo list
+pkgeter repo add --name myrepo --type deb --url https://example.com/debian --release bookworm
+pkgeter repo remove myrepo
+
+# List and apply distribution presets
+pkgeter preset list
+pkgeter preset apply centos-9
+
+# Specify a custom output directory
+pkgeter get -p python3 -o ./my-output
+```
+
+## Output
+
+For Debian mode, all `.deb` files are placed in a `debs/` subdirectory. For RPM mode, all `.rpm` files are placed in a `rpms/` subdirectory. An `install.sh` script is generated that runs `dpkg -i` or `rpm -ivh` in dependency order.
+
+```bash
+# Debian: copy the debs/ directory and install.sh to the target machine, then:
+sudo bash install.sh
+
+# RPM: copy the rpms/ directory and install.sh to the target machine, then:
+sudo bash install.sh
+```
+
+## Configuration
+
+pkgeter stores persistent preferences at `~/.config/pkgeter/config.yaml`. This file is automatically created when you run the tool.
+
+```yaml
+backend: debian
+arch: amd64
+repos:
+  - name: debian-main
+    type: deb
+    url: https://deb.debian.org/debian
+    release: bookworm
+  - name: debian-security
+    type: deb
+    url: https://security.debian.org/debian-security
+    release: bookworm-security
+```
+
+CLI flags override config file values. Apply a preset to quickly populate the config:
+
+```bash
+pkgeter preset apply centos-9
+```
+
+## How It Works
+
+1. **Download package database** — fetches metadata from configured repositories (Packages.gz for Debian, repomd.xml + primary.xml.gz for RPM)
+2. **Parse dependency tree** — recursively resolves all required packages
+3. **Download package files** — downloads each package with SHA256 verification
+4. **Generate output** — creates `debs/` or `rpms/` directory with `install.sh`
+
+## Distribution Presets
+
+| Preset | Backend | Included Repositories |
+|--------|---------|----------------------|
+| `debian-bookworm` | deb | main, security, updates |
+| `debian-bullseye` | deb | main, security, updates |
+| `debian-trixie`  | deb | main, security, updates |
+| `centos-9` | rpm | BaseOS, AppStream, EPEL |
+| `pve8` | deb | bookworm main, security, updates + pve-no-subscription |
+
+## License
+
+MIT

pkgeter-1.1.0/README.md

@@ -0,0 +1,122 @@
+# pkgeter <small>v1.1</small>
+
+**English** | [中文](README_CH.md)
+
+**Offline package downloader** — supports **Debian/apt** and **RPM/dnf** (CentOS Stream). Resolve dependency trees, download `.deb` or `.rpm` files, and generate an offline install script.
+
+Works on any platform (Linux, Windows, macOS) — useful when you need to install packages on an air-gapped or offline machine.
+
+## Features
+
+- **Dual backend** — supports Debian (`dpkg`) and RPM (`rpm`) based distributions
+- **Distribution presets** — one-command selection: `--distro debian-bookworm`, `--distro centos-9`
+- **Interactive REPL** — run `pkgeter` with no arguments to enter a switch-style CLI with prefix matching and TAB completion
+- **Multi-repo merge** — automatically combines repositories (e.g., main + security, BaseOS + AppStream + EPEL)
+- **Dependency resolution** — recursively resolves all dependencies for the target packages
+- **Skip installed packages** — optionally provide a `dpkg -l` output to skip already-installed packages
+- **SHA256 verification** — validates every downloaded `.deb` or `.rpm` file
+- **Source caching** — caches repository metadata with SHA256 validation (like APT), only re-downloads when changed
+- **Offline install script** — auto-generates `install.sh` that runs `dpkg -i` or `rpm -ivh` in dependency order
+- **Multiple mirrors** — specify fallback mirrors, tried in order until one succeeds
+- **Persistent config** — preferences saved to `~/.config/pkgeter/config.yaml`
+- **Repo management** — add, list, and remove custom repositories via `pkgeter repo`
+
+## Installation
+### From source
+
+```bash
+git clone https://github.com/mlzxgzy/pkgeter.git
+cd pkgeter
+pip install -e .
+```
+
+## Usage
+
+```bash
+# Interactive REPL (no arguments)
+pkgeter
+
+# Download packages using a distribution preset
+pkgeter get -p nginx --distro debian-bookworm
+pkgeter get -p nginx --distro centos-9
+pkgeter get -p nginx --distro debian-bullseye
+
+# Short prefix forms work too
+pkgeter g -p nginx --distro centos-9
+
+# Legacy usage (backward compatible)
+pkgeter get -p vim
+pkgeter get -p nginx -r bookworm -a amd64
+
+# Specify multiple mirrors (tried in order)
+pkgeter get -p nginx -m https://deb.debian.org/debian -m https://ftp.debian.org/debian
+
+# Manage repositories
+pkgeter repo list
+pkgeter repo add --name myrepo --type deb --url https://example.com/debian --release bookworm
+pkgeter repo remove myrepo
+
+# List and apply distribution presets
+pkgeter preset list
+pkgeter preset apply centos-9
+
+# Specify a custom output directory
+pkgeter get -p python3 -o ./my-output
+```
+
+## Output
+
+For Debian mode, all `.deb` files are placed in a `debs/` subdirectory. For RPM mode, all `.rpm` files are placed in a `rpms/` subdirectory. An `install.sh` script is generated that runs `dpkg -i` or `rpm -ivh` in dependency order.
+
+```bash
+# Debian: copy the debs/ directory and install.sh to the target machine, then:
+sudo bash install.sh
+
+# RPM: copy the rpms/ directory and install.sh to the target machine, then:
+sudo bash install.sh
+```
+
+## Configuration
+
+pkgeter stores persistent preferences at `~/.config/pkgeter/config.yaml`. This file is automatically created when you run the tool.
+
+```yaml
+backend: debian
+arch: amd64
+repos:
+  - name: debian-main
+    type: deb
+    url: https://deb.debian.org/debian
+    release: bookworm
+  - name: debian-security
+    type: deb
+    url: https://security.debian.org/debian-security
+    release: bookworm-security
+```
+
+CLI flags override config file values. Apply a preset to quickly populate the config:
+
+```bash
+pkgeter preset apply centos-9
+```
+
+## How It Works
+
+1. **Download package database** — fetches metadata from configured repositories (Packages.gz for Debian, repomd.xml + primary.xml.gz for RPM)
+2. **Parse dependency tree** — recursively resolves all required packages
+3. **Download package files** — downloads each package with SHA256 verification
+4. **Generate output** — creates `debs/` or `rpms/` directory with `install.sh`
+
+## Distribution Presets
+
+| Preset | Backend | Included Repositories |
+|--------|---------|----------------------|
+| `debian-bookworm` | deb | main, security, updates |
+| `debian-bullseye` | deb | main, security, updates |
+| `debian-trixie`  | deb | main, security, updates |
+| `centos-9` | rpm | BaseOS, AppStream, EPEL |
+| `pve8` | deb | bookworm main, security, updates + pve-no-subscription |
+
+## License
+
+MIT

pkgeter-1.1.0/pkgeter/__init__.py

@@ -0,0 +1,3 @@
+"""pkgeter - Offline Debian package downloader."""
+
+from pkgeter._version import __version__

pkgeter-1.1.0/pkgeter/__main__.py

@@ -0,0 +1,12 @@
+"""Entry point for `python -m pkgeter` and `pkgeter` CLI command."""
+
+import sys
+
+
+def main():
+    from pkgeter.cli import run_cli
+    run_cli()
+
+
+if __name__ == "__main__":
+    main()

pkgeter-1.1.0/pkgeter/_version.py

@@ -0,0 +1 @@
+__version__ = "1.0.0"

pkgeter-1.1.0/pkgeter/backend/__init__.py

@@ -0,0 +1,69 @@
+"""Backend abstraction for package managers (apt, dnf/yum)."""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+
+from pkgeter.models import PackageInfo, RepoConfig
+
+
+class PmBackend(ABC):
+    """Abstract interface for package manager backends."""
+
+    @property
+    @abstractmethod
+    def name(self) -> str:
+        """Backend identifier, e.g. 'debian', 'rpm'."""
+        ...
+
+    @abstractmethod
+    def download_package_db(
+        self,
+        repos: list[RepoConfig],
+        arch: str,
+        timeout: int = 60,
+        force_update: bool = False,
+    ) -> dict[str, PackageInfo]:
+        """Download metadata from all repos, merge into a single package DB."""
+        ...
+
+    @abstractmethod
+    def build_download_url(self, base_url: str, pkg: PackageInfo) -> str:
+        """Construct the remote download URL for a package file."""
+        ...
+
+    @abstractmethod
+    def generate_install_script(
+        self, files: list[str], target_packages: list[str],
+    ) -> str:
+        """Generate shell script for offline installation."""
+        ...
+
+    def merge_package_dbs(
+        self, dbs: list[dict[str, PackageInfo]],
+    ) -> dict[str, PackageInfo]:
+        """Merge multiple package DBs. Later repos override earlier ones."""
+        merged: dict[str, PackageInfo] = {}
+        for db in dbs:
+            merged.update(db)
+        return merged
+
+    @property
+    def cache(self) -> "PackageCache | None":
+        """Lazily-initialized SQLite package cache."""
+        if not hasattr(self, "_cache"):
+            try:
+                from pkgeter.db.package_cache import PackageCache
+                self._cache: PackageCache | None = PackageCache()
+            except Exception:
+                self._cache = None
+        return self._cache
+
+    @staticmethod
+    def build_source_id(backend_type: str, url: str, release: str, arch: str, component: str = "") -> str:
+        """Build a unique source identifier for the cache."""
+        sanitized = url.removeprefix("https://").removeprefix("http://").rstrip("/")
+        parts = [backend_type, sanitized, release, arch]
+        if component:
+            parts.append(component)
+        return ":".join(parts)

pkgeter-1.1.0/pkgeter/backend/debian.py

@@ -0,0 +1,277 @@
+"""Debian/APT backend implementation.
+
+Implements :class:`PmBackend` for the Debian package manager (``dpkg`` / ``apt``).
+Downloads and caches ``Packages.gz`` metadata from Debian repositories, parses
+stanzas into :class:`PackageInfo` objects, and provides utilities for download
+URL construction and install script generation.
+"""
+
+from __future__ import annotations
+
+import gzip
+import sys
+from typing import Dict
+
+import httpx
+
+from pkgeter.backend import PmBackend
+from pkgeter.db.source_cache import SourceCache
+from pkgeter.models import PackageInfo, RepoConfig, parse_depends_line
+
+
+class DebianBackend(PmBackend):
+    """Debian/APT backend.
+
+    Handles Debian-style ``Packages.gz`` metadata, ``Release``-file-based cache
+    validation, and deterministic ``.deb`` download URLs.
+    """
+
+    # ------------------------------------------------------------------
+    # PmBackend interface
+    # ------------------------------------------------------------------
+
+    @property
+    def name(self) -> str:
+        return "apt"
+
+    def download_package_db(
+        self,
+        repos: list[RepoConfig],
+        arch: str,
+        timeout: int = 60,
+        force_update: bool = False,
+    ) -> Dict[str, PackageInfo]:
+        """Download metadata from all Debian repos, merge into a single package DB.
+
+        For each repository:
+
+        * For the ``main`` component: uses :class:`SourceCache` (Release-based
+          SHA256 caching, same strategy as APT).
+        * For other components (or when the cache is unavailable): falls back
+          to a direct HTTP download.
+
+        Errors on individual repos/components are silently skipped so that a
+        single unavailable repo does not break the whole resolution.
+        """
+        dbs: list[Dict[str, PackageInfo]] = []
+
+        for repo in repos:
+            repo_db = self._download_repo(repo, arch, timeout=timeout, force_update=force_update)
+            if repo_db:
+                dbs.append(repo_db)
+
+        return self.merge_package_dbs(dbs)
+
+    @staticmethod
+    def build_download_url(base_url: str, pkg: PackageInfo) -> str:
+        """Construct the remote download URL for a .deb file.
+
+        For Debian, the ``PackageInfo.filename`` stores the relative path from
+        the mirror root (e.g. ``pool/main/v/vsftpd/vsftpd_3.0.5-3_amd64.deb``).
+        """
+        return f"{base_url.rstrip('/')}/{pkg.filename}"
+
+    @staticmethod
+    def generate_install_script(
+        files: list[str],
+        target_packages: list[str],
+    ) -> str:
+        """Generate a bash install script that uses ``dpkg -i``."""
+        pkg_list = " ".join(target_packages)
+        deb_cmds = "\n".join(
+            f'sudo dpkg -i "{name}"' for name in files
+        )
+        return (
+            "#!/bin/bash\n"
+            "# pkgeter - Offline Debian package installation script\n"
+            f"# Target packages: {pkg_list}\n"
+            "#\n"
+            "# Install packages one by one in dependency order.\n"
+            "#\n"
+            '# Auto-detect sudo availability\n'
+            'if ! command -v sudo >/dev/null 2>&1; then\n'
+            '    sudo() { "$@"; }\n'
+            'fi\n'
+            "\n"
+            'SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"\n'
+            'cd "$SCRIPT_DIR"\n'
+            f"{deb_cmds}\n"
+        )
+
+    # ------------------------------------------------------------------
+    # Stanza / Packages.gz parsing (static for easy re-export by compat)
+    # ------------------------------------------------------------------
+
+    @staticmethod
+    def _parse_deb_stanza(text: str) -> PackageInfo | None:
+        """Parse a single Debian package stanza from a ``Packages`` file.
+
+        Returns ``None`` when the stanza does not contain a ``Package`` field.
+        """
+        text = text.strip()
+        pkg = PackageInfo(package="", version="")
+
+        current_key: str | None = None
+        current_value: list[str] = []
+
+        def _set_field(key: str, value: str) -> None:
+            if key == "Package":
+                pkg.package = value
+            elif key == "Version":
+                pkg.version = value
+            elif key == "Architecture":
+                pkg.arch = value
+            elif key == "Filename":
+                pkg.filename = value
+            elif key == "SHA256":
+                pkg.sha256 = value
+            elif key == "Size":
+                try:
+                    pkg.size = int(value)
+                except ValueError:
+                    pass
+            elif key == "Description":
+                pkg.description = value
+            elif key == "Provides":
+                pkg.provides = [s.strip() for s in value.split(",")]
+            elif key == "Depends":
+                pkg.depends = parse_depends_line(value)
+
+        for line in text.split("\n"):
+            if line.startswith(" ") or line.startswith("\t"):
+                if current_key:
+                    current_value.append(line.strip())
+                continue
+            if ":" in line:
+                if current_key:
+                    _set_field(current_key, " ".join(current_value))
+                current_key = line.split(":", 1)[0].strip()
+                rest = line.split(":", 1)[1].strip()
+                current_value = [rest]
+            else:
+                current_key = None
+                current_value = []
+
+        if current_key and current_value:
+            _set_field(current_key, " ".join(current_value))
+
+        return pkg if pkg.package else None
+
+    @staticmethod
+    def _parse_packages_gz(data: bytes) -> Dict[str, PackageInfo]:
+        """Parse gzip-compressed (or raw) ``Packages`` content into a dict.
+
+        Keys are package names (lowercase), values are :class:`PackageInfo`.
+        """
+        try:
+            raw = gzip.decompress(data)
+        except OSError:
+            raw = data
+
+        text = raw.decode("utf-8", errors="replace")
+        packages: Dict[str, PackageInfo] = {}
+
+        stanzas = text.split("\n\n")
+        for stanza in stanzas:
+            stanza = stanza.strip()
+            if not stanza:
+                continue
+            info = DebianBackend._parse_deb_stanza(stanza)
+            if info and info.package:
+                packages[info.package] = info
+        return packages
+
+    # ------------------------------------------------------------------
+    # Internal helpers
+    # ------------------------------------------------------------------
+
+    def _download_repo(
+        self,
+        repo: RepoConfig,
+        arch: str,
+        *,
+        timeout: int = 60,
+        force_update: bool = False,
+    ) -> Dict[str, PackageInfo] | None:
+        """Download and parse *all components* of a single repository."""
+        components = repo.components or ["main"]
+        repo_db: Dict[str, PackageInfo] = {}
+
+        for component in components:
+            try:
+                parsed = self._download_component(
+                    repo.url, repo.release, component, arch,
+                    timeout=timeout, force_update=force_update,
+                )
+                if parsed:
+                    repo_db.update(parsed)
+            except Exception:
+                # Don't fail entirely just because one component is bad
+                continue
+
+        for pkg in repo_db.values():
+            pkg.base_url = repo.url
+        return repo_db if repo_db else None
+
+    def _download_component(
+        self,
+        mirror: str,
+        release: str,
+        component: str,
+        arch: str,
+        *,
+        timeout: int = 60,
+        force_update: bool = False,
+    ) -> Dict[str, PackageInfo] | None:
+        source_id = self.build_source_id("deb", mirror, release, arch, component)
+
+        if component == "main":
+            cache_obj = SourceCache(mirror, release, arch)
+            if cache_obj.update(timeout=timeout, force_update=force_update):
+                action = cache_obj.last_action
+                if action == "cache_hit":
+                    print(" (cached)", end="", flush=True)
+                elif action == "downloaded":
+                    print(" (downloaded)", end="", flush=True)
+
+                # Check SQLite cache before parsing
+                raw = cache_obj.read_packages_gz()
+                if raw is not None:
+                    file_sha = cache_obj._file_sha256(cache_obj._packages_gz_path)
+                    if file_sha and not force_update and self.cache:
+                        if self.cache.is_fresh(source_id, file_sha):
+                            loaded = self.cache.load(source_id)
+                            if loaded is not None:
+                                return loaded
+                    # Parse and cache
+                    parsed = self._parse_packages_gz(raw)
+                    if file_sha and self.cache:
+                        self.cache.store(source_id, file_sha, parsed)
+                    return parsed
+
+        # Direct HTTP fallback (non-main components, or cache failure)
+        print(" (downloading)", end="", flush=True)
+        url = self._build_component_url(mirror, release, component, arch)
+        with httpx.Client(timeout=timeout) as client:
+            resp = client.get(url, follow_redirects=True)
+            resp.raise_for_status()
+        parsed = self._parse_packages_gz(resp.content)
+
+        # Cache the directly-downloaded result too
+        if self.cache:
+            import hashlib
+            content_sha = hashlib.sha256(resp.content).hexdigest()
+            self.cache.store(source_id, content_sha, parsed)
+
+        return parsed
+
+    @staticmethod
+    def _build_component_url(
+        mirror: str,
+        release: str,
+        component: str,
+        arch: str,
+    ) -> str:
+        """Build the ``Packages.gz`` URL for an arbitrary component."""
+        base = mirror.rstrip("/")
+        return f"{base}/dists/{release}/{component}/binary-{arch}/Packages.gz"