cli-tools-kit 0.6.4__tar.gz → 0.7.1__tar.gz

{cli_tools_kit-0.6.4 → cli_tools_kit-0.7.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cli-tools-kit
-Version: 0.6.4
+Version: 0.7.1
 Summary: Installer protocol + helpers for self-installing Python CLI/GUI tools (desktop shortcuts, bash aliases, cron entries), plus reusable tkinter and curses installer screens
 Author: Steffen Probst
 License: MIT License
@@ -403,6 +403,37 @@ name = "manim-kit"
 url = "https://github.com/AutomatedAlchemy/manim-kit"
 ```
 
+An entry may name a GitHub organisation instead of one repo. The installer lists
+the org's repos, keeps the ones carrying a topic, and turns each into an ordinary
+source, so a new tool in the org appears without anyone editing this file:
+
+```toml
+[[source]]
+org     = "AutomatedAlchemy"
+topic   = "cli-tool-kit"                  # the default when omitted
+exclude = ["alchemy-installer"]           # repo names to skip
+include = ["manim-kit"]                   # allowlist; wins over exclude
+```
+
+The topic decides what is cloned, and the walker plus the `--advertise` probe
+decide what is a tool: a repo that carries the topic but holds no tool clones,
+advertises nothing and is dropped like any other directory. `org` cannot be
+combined with `url` or `path`, and an explicit `[[source]]` with the same `name`
+as a listed repo wins, so one tool can be pinned to a fork or a local checkout
+while the rest of the org follows the listing. Archived repos are left out.
+`include` is an allowlist and overrides `exclude`; the topic is required either
+way. A `path` in `installer.local.toml` pins a listed repo by its name just as it
+pins a tracked source, so a checkout already on the machine is used instead of
+being cloned.
+
+The listing is one `GET` to `api.github.com`, cached for a day under the
+identity's cache directory, and `--refresh` fetches again. With the GitHub CLI
+logged in, its token is used and the org's private repos are listed too; without
+it the public listing is used and nothing is required. When the listing fails the
+cached one is used however old it is, and with no cache at all the directories
+already under the root are used — both say so in one line. `--check` never
+fetches: it reads the cache, or the root.
+
 `installer.local.toml` next to it is optional and belongs to one machine, so keep
 it out of git. It sets the root and replaces a `path` for a source matched by
 `name`:
@@ -442,6 +473,9 @@ colleague without access to a private repo still gets everybody else's tools.
 given by `path` is never pulled. Cloning happens in the engine's `pre_discovery`
 hook, which `--check` skips, so the login check stays network-free.
 
+An `org` entry is the only thing in this module that reaches anything but git,
+it is opt-in per entry, and nothing is fetched when no such entry exists.
+
 A repo that is itself an installer tree can carry its own `installer.toml`. Its
 `[[source]]` entries are resolved too, one nested level deep and no further, with
 paths relative to that file and clones under the same root. A path that is

{cli_tools_kit-0.6.4 → cli_tools_kit-0.7.1}/README.md

@@ -350,6 +350,37 @@ name = "manim-kit"
 url = "https://github.com/AutomatedAlchemy/manim-kit"
 ```
 
+An entry may name a GitHub organisation instead of one repo. The installer lists
+the org's repos, keeps the ones carrying a topic, and turns each into an ordinary
+source, so a new tool in the org appears without anyone editing this file:
+
+```toml
+[[source]]
+org     = "AutomatedAlchemy"
+topic   = "cli-tool-kit"                  # the default when omitted
+exclude = ["alchemy-installer"]           # repo names to skip
+include = ["manim-kit"]                   # allowlist; wins over exclude
+```
+
+The topic decides what is cloned, and the walker plus the `--advertise` probe
+decide what is a tool: a repo that carries the topic but holds no tool clones,
+advertises nothing and is dropped like any other directory. `org` cannot be
+combined with `url` or `path`, and an explicit `[[source]]` with the same `name`
+as a listed repo wins, so one tool can be pinned to a fork or a local checkout
+while the rest of the org follows the listing. Archived repos are left out.
+`include` is an allowlist and overrides `exclude`; the topic is required either
+way. A `path` in `installer.local.toml` pins a listed repo by its name just as it
+pins a tracked source, so a checkout already on the machine is used instead of
+being cloned.
+
+The listing is one `GET` to `api.github.com`, cached for a day under the
+identity's cache directory, and `--refresh` fetches again. With the GitHub CLI
+logged in, its token is used and the org's private repos are listed too; without
+it the public listing is used and nothing is required. When the listing fails the
+cached one is used however old it is, and with no cache at all the directories
+already under the root are used — both say so in one line. `--check` never
+fetches: it reads the cache, or the root.
+
 `installer.local.toml` next to it is optional and belongs to one machine, so keep
 it out of git. It sets the root and replaces a `path` for a source matched by
 `name`:
@@ -389,6 +420,9 @@ colleague without access to a private repo still gets everybody else's tools.
 given by `path` is never pulled. Cloning happens in the engine's `pre_discovery`
 hook, which `--check` skips, so the login check stays network-free.
 
+An `org` entry is the only thing in this module that reaches anything but git,
+it is opt-in per entry, and nothing is fetched when no such entry exists.
+
 A repo that is itself an installer tree can carry its own `installer.toml`. Its
 `[[source]]` entries are resolved too, one nested level deep and no further, with
 paths relative to that file and clones under the same root. A path that is

{cli_tools_kit-0.6.4 → cli_tools_kit-0.7.1}/cli_tools_kit/__init__.py

@@ -56,4 +56,4 @@ __all__ = [
     "read_installed_skill",
 ]
 
-__version__ = "0.6.1"
+__version__ = "0.7.1"

{cli_tools_kit-0.6.4 → cli_tools_kit-0.7.1}/cli_tools_kit/sources.py

@@ -40,6 +40,21 @@ are resolved too, one nested level deep and no further. Paths in a nested file
 are relative to that file, clones still go under the same root, a path already
 resolved is not visited twice, and duplicates are dropped.
 
+An entry may name a GitHub organisation instead of one repo. The installer
+lists the org's repos, keeps the ones carrying a topic, and turns each into an
+ordinary source, so everything after that step is unchanged:
+
+    [[source]]
+    org     = "AutomatedAlchemy"
+    topic   = "cli-tool-kit"                  # the default when omitted
+    exclude = ["alchemy-installer"]           # repo names to skip
+    include = ["manim-kit"]                   # allowlist; wins over exclude
+
+``org`` is mutually exclusive with ``url`` and ``path``. The listing is cached
+for a day, a network error falls back to the cached list and then to the
+directories already under the root, and an explicit ``[[source]]`` with the
+same ``name`` always wins over an org-derived one.
+
 The whole feature is three calls:
 
     sources = load_sources("installer.toml")
@@ -54,15 +69,22 @@ or, for a wrapper that just wants the installer:
 
 from __future__ import annotations
 
+import json
 import os
+import re
+import shutil
 import subprocess
 import sys
+import time
+import urllib.error
+import urllib.request
 from dataclasses import dataclass
 from pathlib import Path
-from typing import Callable, List, Optional, Sequence
+from typing import Callable, List, Optional, Sequence, Tuple
 
-__all__ = ["Source", "load_sources", "resolve_sources", "run_installer",
-           "local_root", "save_local_root", "default_root"]
+__all__ = ["Source", "OrgSource", "load_sources", "expand_org_sources",
+           "resolve_sources", "run_installer", "local_root", "save_local_root",
+           "default_root"]
 
 # How far below the top-level installer.toml a nested one is still read.
 MAX_NESTING = 1
@@ -87,6 +109,53 @@ class Source:
     path: Optional[str] = None
 
 
+# The default GitHub topic an org's repos are tagged with to be offered.
+DEFAULT_ORG_TOPIC = "cli-tool-kit"
+
+# A GitHub org or user name, and a topic: both are pasted into a URL path, so
+# they stay to the characters GitHub itself allows. \Z, not $, so a trailing
+# newline cannot smuggle a second path segment in.
+_ORG_RE = re.compile(r"^[A-Za-z0-9-]{1,39}\Z")
+_TOPIC_RE = re.compile(r"^[A-Za-z0-9-]{1,50}\Z")
+
+# The one host this module talks to, hard-coded so a config file cannot point
+# the listing at somewhere else.
+GITHUB_API = "https://api.github.com"
+
+# How long a cached listing is used without asking GitHub again.
+ORG_CACHE_TTL = 24 * 60 * 60
+
+# Enough for 500 repos; a listing longer than that is a config mistake.
+ORG_MAX_PAGES = 5
+
+ORG_TIMEOUT = 10
+GH_TOKEN_TIMEOUT = 5
+
+
+@dataclass(frozen=True)
+class OrgSource:
+    """One GitHub organisation whose topic-tagged repos become sources.
+
+    Expanded into ordinary :class:`Source` entries by
+    :func:`expand_org_sources`, which is the only place in this module that
+    reaches the network.
+    """
+
+    org: str
+    topic: str = DEFAULT_ORG_TOPIC
+    include: Tuple[str, ...] = ()
+    exclude: Tuple[str, ...] = ()
+    # ``installer.local.toml``'s ``[[source]]`` path overrides, as already
+    # absolute paths keyed by repo name. A listed repo whose name is in here
+    # resolves to that checkout instead of being cloned, exactly as the
+    # override works for a tracked ``[[source]]``. A tuple of pairs, not a
+    # dict, because this dataclass is frozen and has to stay hashable.
+    pins: Tuple[Tuple[str, str], ...] = ()
+
+    def pin_for(self, name: str) -> Optional[str]:
+        return dict(self.pins).get(name)
+
+
 # --- TOML -------------------------------------------------------------------
 
 def _toml_module():
@@ -175,12 +244,51 @@ def save_local_root(config_path, root, local_path=None, log: Callable = print) -
     return True
 
 
-def load_sources(config_path, local_path=None, log: Callable = print) -> List[Source]:
+def _names(entry: dict, key: str) -> Tuple[str, ...]:
+    """One of the ``include`` / ``exclude`` lists, as a tuple of strings."""
+    raw = entry.get(key)
+    if not isinstance(raw, list):
+        return ()
+    return tuple(item for item in raw if isinstance(item, str) and item)
+
+
+def _org_source(entry: dict, config_name: str, log: Callable,
+                pins: Tuple[Tuple[str, str], ...] = ()) -> Optional[OrgSource]:
+    """One ``[[source]]`` table with an ``org``, or None when it is unusable.
+
+    Reported and dropped the same way an https-only violation is: one line
+    naming what is wrong, and the other sources still install.
+    """
+    org = entry.get("org")
+    if not isinstance(org, str) or not _ORG_RE.match(org):
+        log(f"{config_name}: a [[source]] with an unusable org "
+            f"({org!r}), skipped")
+        return None
+    if entry.get("url") or entry.get("path"):
+        log(f"{org}: a [[source]] cannot have both org and url/path, skipped")
+        return None
+    topic = entry.get("topic", DEFAULT_ORG_TOPIC)
+    if not isinstance(topic, str) or not _TOPIC_RE.match(topic):
+        log(f"{org}: {topic!r} is not a usable topic, skipped")
+        return None
+    return OrgSource(org=org, topic=topic,
+                     include=_names(entry, "include"),
+                     exclude=_names(entry, "exclude"),
+                     pins=pins)
+
+
+def load_sources(config_path, local_path=None, log: Callable = print) -> List:
     """The ``[[source]]`` entries of one TOML file, local overrides applied.
 
     ``local_path`` defaults to ``installer.local.toml`` next to ``config_path``.
     A ``path`` is taken relative to the file it is written in. An entry without
     a name is reported and dropped.
+
+    An entry that carries an ``org`` instead of a ``name`` becomes an
+    :class:`OrgSource` in the returned list. :func:`expand_org_sources` turns
+    those into ordinary :class:`Source` entries; :func:`resolve_sources` ignores
+    any that are left, so a caller that does not expand simply gets no tools
+    from the org rather than an error.
     """
     config_path = Path(config_path)
     local_path = Path(local_path) if local_path is not None else _local_path_for(config_path)
@@ -191,14 +299,29 @@ def load_sources(config_path, local_path=None, log: Callable = print) -> List[So
                  for entry in local.get("source") or []
                  if isinstance(entry, dict) and entry.get("name")}
 
-    sources: List[Source] = []
+    # The same overrides an explicit [[source]] gets, kept for the org
+    # expansion: the repos it derives do not exist yet at this point, so a pin
+    # naming one of them can only be applied later, by name.
+    pins = tuple((name, _absolute(local_path.parent, entry["path"]))
+                 for name, entry in overrides.items()
+                 if isinstance(entry.get("path"), str) and entry["path"])
+
+    sources: List = []
     for entry in data.get("source") or []:
         if not isinstance(entry, dict):
             continue
         name = entry.get("name")
         if not name:
+            if "org" in entry:
+                org_source = _org_source(entry, config_path.name, log, pins)
+                if org_source is not None:
+                    sources.append(org_source)
+                continue
             log(f"{config_path.name}: a [[source]] without a name, skipped")
             continue
+        if entry.get("org"):
+            log(f"{name}: a [[source]] cannot have both org and url/path, skipped")
+            continue
         override = overrides.get(name, {}).get("path")
         raw = override or entry.get("path")
         base = local_path.parent if override else config_path.parent
@@ -207,6 +330,253 @@ def load_sources(config_path, local_path=None, log: Callable = print) -> List[So
     return sources
 
 
+# --- GitHub org listings ----------------------------------------------------
+
+def _gh_token() -> Optional[str]:
+    """The token ``gh auth token`` prints, or None.
+
+    Opportunistic: with the GitHub CLI logged in, the listing also sees the
+    org's private repos. Without it the public listing is used. The token is
+    never logged.
+    """
+    if not shutil.which("gh"):
+        return None
+    try:
+        result = subprocess.run(["gh", "auth", "token"], capture_output=True,
+                                text=True, timeout=GH_TOKEN_TIMEOUT)
+    except (OSError, subprocess.SubprocessError):
+        return None
+    if result.returncode != 0:
+        return None
+    token = result.stdout.strip()
+    return token or None
+
+
+def _api_version() -> str:
+    from . import __version__  # noqa: PLC0415 — avoids an import cycle at module load
+    return __version__
+
+
+def _fetch_org_repos(org: str, token: Optional[str]) -> List[dict]:
+    """Every repo of one org, over as many pages as GitHub needs.
+
+    Raises ``OSError`` (which ``urllib`` errors are) on anything that goes
+    wrong, so the one caller can fall back in a single place.
+    """
+    headers = {"Accept": "application/vnd.github+json",
+               "User-Agent": f"cli-tools-kit/{_api_version()}"}
+    if token:
+        headers["Authorization"] = f"Bearer {token}"
+    repos: List[dict] = []
+    for page in range(1, ORG_MAX_PAGES + 1):
+        query = f"per_page=100&page={page}"
+        if not token:
+            # Without a token only public repos are visible anyway; asking for
+            # them explicitly keeps the response small.
+            query += "&type=public"
+        request = urllib.request.Request(  # noqa: S310 — the host is hard-coded above
+            f"{GITHUB_API}/orgs/{org}/repos?{query}", headers=headers)
+        with urllib.request.urlopen(request, timeout=ORG_TIMEOUT) as response:
+            batch = json.loads(response.read().decode("utf-8"))
+        if not isinstance(batch, list):
+            raise OSError("the listing was not a JSON array")
+        repos.extend(item for item in batch if isinstance(item, dict))
+        if len(batch) < 100:
+            break
+    return repos
+
+
+def _keep_fields(repos: Sequence[dict]) -> List[dict]:
+    """Only the fields this module uses, so the cache stays small and readable."""
+    keep = ("name", "clone_url", "topics", "archived", "default_branch",
+            "description")
+    return [{field_name: repo.get(field_name) for field_name in keep}
+            for repo in repos if repo.get("name")]
+
+
+def _cache_file(cache_dir, org: str) -> Path:
+    return Path(os.path.expanduser(str(cache_dir))) / f"org-{org}.json"
+
+
+def _read_org_cache(cache_dir, org: str) -> Optional[dict]:
+    """The cached listing for one org, or None when there is none to read."""
+    try:
+        with open(_cache_file(cache_dir, org), encoding="utf-8") as fh:
+            cached = json.load(fh)
+    except (OSError, ValueError):
+        return None
+    if not isinstance(cached, dict) or not isinstance(cached.get("repos"), list):
+        return None
+    return cached
+
+
+def _write_org_cache(cache_dir, org: str, topic: str, repos: Sequence[dict],
+                     log: Callable) -> None:
+    path = _cache_file(cache_dir, org)
+    try:
+        path.parent.mkdir(parents=True, exist_ok=True)
+        path.write_text(json.dumps({"fetched_at": time.time(), "topic": topic,
+                                    "repos": list(repos)}, indent=1),
+                        encoding="utf-8")
+    except OSError as exc:
+        log(f"{org}: the listing was not cached ({exc})")
+
+
+def _age(seconds: float) -> str:
+    """"2h ago", "3 days ago" — how long ago a listing was fetched."""
+    delta = max(0.0, time.time() - seconds)
+    if delta < 90 * 60:
+        return f"{int(delta // 60)}min ago"
+    if delta < 36 * 3600:
+        return f"{int(delta // 3600)}h ago"
+    return f"{int(delta // 86400)} days ago"
+
+
+def _selected(entry: OrgSource, repos: Sequence[dict]) -> List[dict]:
+    """The repos of one listing this entry offers.
+
+    ``include`` is an allowlist and wins over ``exclude``; the topic is still
+    required either way, so a repo that lost its tag stops being offered
+    without anyone having to edit the config.
+    """
+    chosen = []
+    for repo in repos:
+        name = repo.get("name")
+        if not name or repo.get("archived"):
+            continue
+        topics = repo.get("topics") or []
+        if entry.topic not in topics:
+            continue
+        if entry.include:
+            if name not in entry.include:
+                continue
+        elif name in entry.exclude:
+            continue
+        chosen.append(repo)
+    return chosen
+
+
+def _from_root(entry: OrgSource, root, log: Callable) -> List[Source]:
+    """The last fallback: the org's checkouts that are already under the root.
+
+    No listing and no cache, so what is on disk is all this run knows about.
+    A path-only source, because without a listing there is no clone URL.
+    """
+    base = Path(os.path.expanduser(str(root)))
+    found = []
+    try:
+        entries = sorted(item for item in base.iterdir() if item.is_dir())
+    except OSError:
+        entries = []
+    for item in entries:
+        if entry.include:
+            if item.name not in entry.include:
+                continue
+        elif item.name in entry.exclude:
+            continue
+        found.append(Source(name=item.name, path=str(item.resolve())))
+    log(f"{entry.org}: no listing and no cache, using the "
+        f"{len(found)} checkout(s) already under {base}")
+    return found
+
+
+def _org_listing(entry: OrgSource, *, refresh: bool, clone: bool, cache_dir,
+                 log: Callable) -> Optional[Tuple[List[dict], str]]:
+    """One org's repo list plus the line describing where it came from.
+
+    None when neither the network nor the cache produced one. ``clone=False``
+    is the network-free path, so it never fetches.
+    """
+    cached = _read_org_cache(cache_dir, entry.org)
+    fetched_at = cached.get("fetched_at") if cached else None
+    fresh_enough = (isinstance(fetched_at, (int, float))
+                    and time.time() - fetched_at < ORG_CACHE_TTL)
+
+    if not clone:
+        if cached is None:
+            return None
+        return cached["repos"], f"cached listing from {_age(fetched_at or 0)}"
+    if cached is not None and fresh_enough and not refresh:
+        return cached["repos"], f"listed {_age(fetched_at)}"
+
+    try:
+        repos = _keep_fields(_fetch_org_repos(entry.org, _gh_token()))
+    except (OSError, ValueError, urllib.error.HTTPError) as exc:
+        reason = getattr(exc, "reason", None) or exc
+        if cached is None:
+            return None
+        log(f"{entry.org}: not listed, using the cached listing from "
+            f"{_age(fetched_at or 0)} ({reason})")
+        return cached["repos"], f"cached listing from {_age(fetched_at or 0)}"
+
+    if cached is not None:
+        before = {repo.get("name") for repo in _selected(entry, cached["repos"])}
+        gone = sorted(before - {repo.get("name") for repo in _selected(entry, repos)})
+        if gone:
+            log(f"{entry.org}: {len(gone)} repos dropped since the last "
+                f"listing: {', '.join(gone)}")
+    _write_org_cache(cache_dir, entry.org, entry.topic, repos, log)
+    return repos, "listed just now"
+
+
+def expand_org_sources(sources: Sequence, *, refresh: bool = False, cache_dir,
+                       root=None, clone: bool = True,
+                       log: Callable = print) -> List[Source]:
+    """Replace every :class:`OrgSource` with the repos it stands for.
+
+    Each kept repo becomes an ordinary ``Source(name=<repo>, url=<clone_url>)``,
+    so resolution, cloning, nesting, discovery and the ``--advertise`` probe all
+    work on it unchanged. A repo that carries the topic but turns out to hold no
+    tool clones, advertises nothing, and is dropped by the walker as any other
+    directory is.
+
+    An explicit ``[[source]]`` with the same ``name`` wins, so one repo can be
+    pinned to a fork or a local checkout while the rest of the org follows the
+    listing. A ``path`` in ``installer.local.toml`` naming a listed repo pins it
+    the same way, carried here on ``OrgSource.pins`` because the repo it names
+    does not exist yet when that file is read.
+
+    ``clone=False`` is the network-free path the login check takes: it uses the
+    cache, then the directories already under ``root``, and never fetches.
+    """
+    explicit = {source.name for source in sources if isinstance(source, Source)}
+    expanded: List[Source] = []
+    for source in sources:
+        if isinstance(source, Source):
+            expanded.append(source)
+            continue
+        if not isinstance(source, OrgSource):
+            continue
+        listing = _org_listing(source, refresh=refresh, clone=clone,
+                               cache_dir=cache_dir, log=log)
+        if listing is None:
+            derived = _from_root(source, root, log) if root is not None else []
+        else:
+            repos, provenance = listing
+            kept = _selected(source, repos)
+            log(f"{source.org}: {len(kept)} repos tagged {source.topic} "
+                f"({provenance})")
+            derived = [Source(name=repo["name"], url=repo.get("clone_url"))
+                       for repo in kept]
+        # A local pin wins over the clone URL, so the checkout on this machine
+        # is used and nothing is fetched. A pin whose directory is gone is left
+        # in place: _resolve_one falls through to the URL, which is what the
+        # override does for a tracked source too.
+        derived = [Source(name=candidate.name, url=candidate.url,
+                          path=source.pin_for(candidate.name) or candidate.path)
+                   for candidate in derived]
+        for candidate in derived:
+            if candidate.name in explicit:
+                continue
+            url = candidate.url
+            if url is not None and not str(url).lower().startswith("https://"):
+                log(f"{candidate.name}: {url} is not an https:// URL, skipped")
+                continue
+            explicit.add(candidate.name)
+            expanded.append(candidate)
+    return expanded
+
+
 # --- resolution -------------------------------------------------------------
 
 def _git(*args):
@@ -273,6 +643,10 @@ def _resolve_level(sources: Sequence[Source], root: Path, refresh: bool, log: Ca
                    clone: bool, config_name: str, depth: int,
                    found: List[Path], seen: set) -> None:
     for source in sources:
+        if not isinstance(source, Source):
+            # An OrgSource nobody expanded. Ignored rather than fatal, so a
+            # caller that skipped expand_org_sources still installs the rest.
+            continue
         path = _resolve_one(source, root, refresh, log, clone)
         if path is None or path in seen:
             continue
@@ -466,6 +840,12 @@ def run_installer(config_path, argv=None, default_root_name: str = "tools", **ru
     pre-discovery hook, which the engine skips on the ``--check`` path, so that
     check stays network-free and sees whatever is already on disk.
 
+    An ``org`` entry in the sources file is expanded into one source per
+    topic-tagged repo before resolution, using the identity's cache directory
+    for the listing. That expansion is the only network call this module makes
+    besides git, it happens only when such an entry exists, and on the
+    ``--check`` path it reads the cache instead of GitHub.
+
     ``default_root_name`` is the folder name the suggestion ends in, so an
     organisation's installer can suggest ``<cwd>/WW3-tools`` rather than
     ``<cwd>/tools``.
@@ -490,15 +870,24 @@ def run_installer(config_path, argv=None, default_root_name: str = "tools", **ru
     except OSError as exc:
         print(f"{root}: not created ({exc})")
     sources = load_sources(config_path)
+    from .identity import LEGACY_IDENTITY  # noqa: PLC0415 — keeps the import graph flat
+    identity = run_kwargs.get("identity") or LEGACY_IDENTITY
+    cache_dir = identity.cache_path
 
     # The engine reads DISCOVERY_ROOTS after the hook has run, so the hook fills
     # this list in place with what it resolved. It is pre-filled with what is on
-    # disk already, for the --check path that never calls the hook.
-    roots = [str(p) for p in resolve_sources(sources, root, clone=False,
-                                             log=lambda *_: None)]
+    # disk already, for the --check path that never calls the hook — which is
+    # why the expansion here is the network-free one.
+    quiet = lambda *_: None  # noqa: E731
+    roots = [str(p) for p in resolve_sources(
+        expand_org_sources(sources, cache_dir=cache_dir, root=root, clone=False,
+                           log=quiet),
+        root, clone=False, log=quiet)]
 
     def pre_discovery(refresh):
-        roots[:] = [str(p) for p in resolve_sources(sources, root, refresh=refresh)]
+        expanded = expand_org_sources(sources, refresh=refresh, cache_dir=cache_dir,
+                                      root=root)
+        roots[:] = [str(p) for p in resolve_sources(expanded, root, refresh=refresh)]
 
     from . import gui_installer  # noqa: PLC0415 — imports tkinter, keep it lazy
     run_kwargs.setdefault("root_dir", root)

{cli_tools_kit-0.6.4 → cli_tools_kit-0.7.1}/cli_tools_kit.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cli-tools-kit
-Version: 0.6.4
+Version: 0.7.1
 Summary: Installer protocol + helpers for self-installing Python CLI/GUI tools (desktop shortcuts, bash aliases, cron entries), plus reusable tkinter and curses installer screens
 Author: Steffen Probst
 License: MIT License
@@ -403,6 +403,37 @@ name = "manim-kit"
 url = "https://github.com/AutomatedAlchemy/manim-kit"
 ```
 
+An entry may name a GitHub organisation instead of one repo. The installer lists
+the org's repos, keeps the ones carrying a topic, and turns each into an ordinary
+source, so a new tool in the org appears without anyone editing this file:
+
+```toml
+[[source]]
+org     = "AutomatedAlchemy"
+topic   = "cli-tool-kit"                  # the default when omitted
+exclude = ["alchemy-installer"]           # repo names to skip
+include = ["manim-kit"]                   # allowlist; wins over exclude
+```
+
+The topic decides what is cloned, and the walker plus the `--advertise` probe
+decide what is a tool: a repo that carries the topic but holds no tool clones,
+advertises nothing and is dropped like any other directory. `org` cannot be
+combined with `url` or `path`, and an explicit `[[source]]` with the same `name`
+as a listed repo wins, so one tool can be pinned to a fork or a local checkout
+while the rest of the org follows the listing. Archived repos are left out.
+`include` is an allowlist and overrides `exclude`; the topic is required either
+way. A `path` in `installer.local.toml` pins a listed repo by its name just as it
+pins a tracked source, so a checkout already on the machine is used instead of
+being cloned.
+
+The listing is one `GET` to `api.github.com`, cached for a day under the
+identity's cache directory, and `--refresh` fetches again. With the GitHub CLI
+logged in, its token is used and the org's private repos are listed too; without
+it the public listing is used and nothing is required. When the listing fails the
+cached one is used however old it is, and with no cache at all the directories
+already under the root are used — both say so in one line. `--check` never
+fetches: it reads the cache, or the root.
+
 `installer.local.toml` next to it is optional and belongs to one machine, so keep
 it out of git. It sets the root and replaces a `path` for a source matched by
 `name`:
@@ -442,6 +473,9 @@ colleague without access to a private repo still gets everybody else's tools.
 given by `path` is never pulled. Cloning happens in the engine's `pre_discovery`
 hook, which `--check` skips, so the login check stays network-free.
 
+An `org` entry is the only thing in this module that reaches anything but git,
+it is opt-in per entry, and nothing is fetched when no such entry exists.
+
 A repo that is itself an installer tree can carry its own `installer.toml`. Its
 `[[source]]` entries are resolved too, one nested level deep and no further, with
 paths relative to that file and clones under the same root. A path that is

{cli_tools_kit-0.6.4 → cli_tools_kit-0.7.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "cli-tools-kit"
-version = "0.6.4"
+version = "0.7.1"
 description = "Installer protocol + helpers for self-installing Python CLI/GUI tools (desktop shortcuts, bash aliases, cron entries), plus reusable tkinter and curses installer screens"
 readme = "README.md"
 requires-python = ">=3.10"

{cli_tools_kit-0.6.4 → cli_tools_kit-0.7.1}/tests/test_sources.py

@@ -523,3 +523,442 @@ def test_run_installer_refuses_to_have_its_own_arguments_overridden(tmp_path: Pa
         sources.run_installer(config, argv=[], discovery_roots=["/x"])
     with pytest.raises(TypeError):
         sources.run_installer(config, argv=[], pre_discovery=lambda refresh: None)
+
+
+# --- GitHub org sources -----------------------------------------------------
+
+def _repo_json(name: str, topics=("cli-tool-kit",), archived: bool = False,
+               scheme: str = "https") -> dict:
+    return {"name": name, "clone_url": f"{scheme}://github.com/acme/{name}.git",
+            "topics": list(topics), "archived": archived,
+            "default_branch": "main", "description": f"the {name} tool"}
+
+
+def _fake_urlopen(pages, calls: list):
+    """A urlopen stand-in serving canned pages of JSON.
+
+    ``pages`` is a list of repo lists, one per requested page; anything past
+    the end is served as an empty page, which is how GitHub ends a listing.
+    """
+    import io
+    import json as _json
+
+    class Response(io.BytesIO):
+        def __enter__(self):
+            return self
+
+        def __exit__(self, *_):
+            self.close()
+            return False
+
+    def urlopen(request, timeout=None):
+        calls.append(request.full_url)
+        page = 1
+        for part in request.full_url.split("?")[-1].split("&"):
+            if part.startswith("page="):
+                page = int(part[len("page="):])
+        body = pages[page - 1] if page <= len(pages) else []
+        return Response(_json.dumps(body).encode("utf-8"))
+
+    return urlopen
+
+
+@pytest.fixture
+def no_gh(monkeypatch):
+    """No GitHub CLI on this host, so the listing stays anonymous."""
+    monkeypatch.setattr(sources.shutil, "which", lambda name: None)
+
+
+@pytest.fixture
+def api(monkeypatch, no_gh):
+    """Serve canned listings and record the URLs that were requested."""
+    calls: list = []
+
+    def serve(*pages):
+        monkeypatch.setattr(sources.urllib.request, "urlopen",
+                            _fake_urlopen(list(pages), calls))
+        return calls
+
+    return serve
+
+
+def _expand(tmp_path: Path, entry, lines=None, **kwargs):
+    return sources.expand_org_sources(
+        [entry], cache_dir=tmp_path / "cache",
+        log=(lines.append if lines is not None else (lambda *_: None)), **kwargs)
+
+
+def test_only_topic_tagged_repos_become_sources(tmp_path: Path, api) -> None:
+    api([_repo_json("kept"), _repo_json("other", topics=("website",))])
+    expanded = _expand(tmp_path, sources.OrgSource(org="acme"))
+    assert [(s.name, s.url) for s in expanded] == [
+        ("kept", "https://github.com/acme/kept.git")]
+
+
+def test_archived_repos_are_left_out(tmp_path: Path, api) -> None:
+    api([_repo_json("live"), _repo_json("old", archived=True)])
+    assert [s.name for s in _expand(tmp_path, sources.OrgSource(org="acme"))] == ["live"]
+
+
+def test_a_listing_is_read_across_pages(tmp_path: Path, api) -> None:
+    first = [_repo_json(f"tool{i:02d}") for i in range(100)]
+    calls = api(first, [_repo_json("last")])
+    expanded = _expand(tmp_path, sources.OrgSource(org="acme"))
+    assert len(expanded) == 101
+    assert expanded[-1].name == "last"
+    assert len(calls) == 2 and "page=2" in calls[1]
+    assert all(call.startswith("https://api.github.com/orgs/acme/repos?") for call in calls)
+
+
+def test_an_http_only_clone_url_is_refused(tmp_path: Path, api) -> None:
+    api([_repo_json("plain", scheme="http")])
+    lines: list = []
+    assert _expand(tmp_path, sources.OrgSource(org="acme"), lines) == []
+    assert any("not an https:// URL" in line for line in lines)
+
+
+def test_a_cached_listing_within_the_ttl_makes_no_call(tmp_path: Path, api) -> None:
+    calls = api([_repo_json("kept")])
+    entry = sources.OrgSource(org="acme")
+    assert len(_expand(tmp_path, entry)) == 1
+    assert len(calls) == 1
+    assert [s.name for s in _expand(tmp_path, entry)] == ["kept"]
+    assert len(calls) == 1                     # the cache answered the second time
+
+
+def test_refresh_fetches_although_the_cache_is_fresh(tmp_path: Path, api) -> None:
+    calls = api([_repo_json("kept")])
+    entry = sources.OrgSource(org="acme")
+    _expand(tmp_path, entry)
+    _expand(tmp_path, entry, refresh=True)
+    assert len(calls) == 2
+
+
+def test_a_stale_cache_is_used_when_the_listing_fails(tmp_path: Path, api,
+                                                      monkeypatch) -> None:
+    api([_repo_json("kept")])
+    entry = sources.OrgSource(org="acme")
+    _expand(tmp_path, entry)
+    cache = tmp_path / "cache" / "org-acme.json"
+    payload = sources.json.loads(cache.read_text(encoding="utf-8"))
+    payload["fetched_at"] -= 10 * 24 * 3600            # ten days old
+    cache.write_text(sources.json.dumps(payload), encoding="utf-8")
+
+    def boom(request, timeout=None):
+        raise sources.urllib.error.URLError("no route to host")
+
+    monkeypatch.setattr(sources.urllib.request, "urlopen", boom)
+    lines: list = []
+    assert [s.name for s in _expand(tmp_path, entry, lines)] == ["kept"]
+    assert any("not listed, using the cached listing from" in line for line in lines)
+
+
+def test_no_cache_and_no_network_falls_back_to_the_root(tmp_path: Path,
+                                                        monkeypatch, no_gh) -> None:
+    root = tmp_path / "root"
+    _repo(root / "on-disk")
+    _repo(root / "skipped")
+
+    def boom(request, timeout=None):
+        raise sources.urllib.error.URLError("no route to host")
+
+    monkeypatch.setattr(sources.urllib.request, "urlopen", boom)
+    lines: list = []
+    expanded = _expand(tmp_path, sources.OrgSource(org="acme", exclude=("skipped",)),
+                       lines, root=root)
+    assert [(s.name, s.path) for s in expanded] == [
+        ("on-disk", str((root / "on-disk").resolve()))]
+    assert any("no listing and no cache" in line for line in lines)
+
+
+def test_the_root_fallback_cannot_tell_which_directories_are_the_orgs(
+        tmp_path: Path, monkeypatch, no_gh) -> None:
+    """Without a listing there is nothing to match names against.
+
+    So the last resort offers every directory under the root that ``exclude``
+    does not name. A directory that holds no tool advertises nothing and the
+    walker drops it, which is what keeps this safe rather than clever.
+    """
+    root = tmp_path / "root"
+    _repo(root / "a-tool")
+    (root / "not-a-repo").mkdir(parents=True)
+
+    def boom(request, timeout=None):
+        raise sources.urllib.error.URLError("no route to host")
+
+    monkeypatch.setattr(sources.urllib.request, "urlopen", boom)
+    expanded = _expand(tmp_path, sources.OrgSource(org="acme"), root=root)
+    assert [s.name for s in expanded] == ["a-tool", "not-a-repo"]
+
+
+def test_the_network_free_path_never_fetches(tmp_path: Path, monkeypatch,
+                                             no_gh) -> None:
+    root = tmp_path / "root"
+    _repo(root / "on-disk")
+
+    def refuse(request, timeout=None):
+        raise AssertionError("clone=False reached the network")
+
+    monkeypatch.setattr(sources.urllib.request, "urlopen", refuse)
+    expanded = _expand(tmp_path, sources.OrgSource(org="acme"), root=root, clone=False)
+    assert [s.name for s in expanded] == ["on-disk"]
+
+
+def test_clone_false_prefers_the_cache_over_the_root(tmp_path: Path, api,
+                                                     monkeypatch) -> None:
+    root = tmp_path / "root"
+    _repo(root / "on-disk")
+    entry = sources.OrgSource(org="acme")
+    api([_repo_json("kept")])
+    _expand(tmp_path, entry)                   # fills the cache
+
+    def refuse(request, timeout=None):
+        raise AssertionError("clone=False reached the network")
+
+    monkeypatch.setattr(sources.urllib.request, "urlopen", refuse)
+    assert [s.name for s in _expand(tmp_path, entry, root=root, clone=False)] == ["kept"]
+
+
+def test_an_explicit_source_wins_over_the_org_listing(tmp_path: Path, api) -> None:
+    api([_repo_json("kept"), _repo_json("pinned")])
+    fork = _repo(tmp_path / "fork")
+    expanded = sources.expand_org_sources(
+        [Source(name="pinned", path=str(fork)), sources.OrgSource(org="acme")],
+        cache_dir=tmp_path / "cache", log=lambda *_: None)
+    assert [(s.name, s.path, s.url) for s in expanded] == [
+        ("pinned", str(fork), None),
+        ("kept", None, "https://github.com/acme/kept.git")]
+
+
+def test_exclude_drops_a_repo_and_include_is_an_allowlist(tmp_path: Path, api) -> None:
+    listing = [_repo_json("a"), _repo_json("b"), _repo_json("c"),
+               _repo_json("untagged", topics=("website",))]
+    api(listing)
+    assert [s.name for s in _expand(tmp_path, sources.OrgSource(org="acme",
+                                                                exclude=("b",)))] \
+        == ["a", "c"]
+    # include wins over exclude, and the topic is still required.
+    assert [s.name for s in _expand(
+        tmp_path / "second", sources.OrgSource(org="acme", include=("b", "untagged"),
+                                               exclude=("b",)))] == ["b"]
+
+
+def test_the_list_header_names_the_count_the_topic_and_the_age(tmp_path: Path,
+                                                               api) -> None:
+    api([_repo_json("a"), _repo_json("b")])
+    lines: list = []
+    _expand(tmp_path, sources.OrgSource(org="acme"), lines)
+    assert lines[0] == "acme: 2 repos tagged cli-tool-kit (listed just now)"
+
+
+def test_a_shrinking_listing_is_reported(tmp_path: Path, api, monkeypatch) -> None:
+    calls = api([_repo_json("a"), _repo_json("b"), _repo_json("c")])
+    entry = sources.OrgSource(org="acme")
+    _expand(tmp_path, entry)
+    monkeypatch.setattr(sources.urllib.request, "urlopen",
+                        _fake_urlopen([[_repo_json("a")]], calls))
+    lines: list = []
+    _expand(tmp_path, entry, lines, refresh=True)
+    assert any("2 repos dropped since the last listing: b, c" in line for line in lines)
+
+
+def test_a_token_from_the_gh_cli_is_sent_and_never_logged(tmp_path: Path,
+                                                          monkeypatch) -> None:
+    calls: list = []
+    monkeypatch.setattr(sources.shutil, "which", lambda name: "/usr/bin/gh")
+
+    class Result:
+        returncode = 0
+        stdout = "gho_secret\n"
+        stderr = ""
+
+    monkeypatch.setattr(sources.subprocess, "run", lambda cmd, **kw: Result())
+    headers: list = []
+    inner = _fake_urlopen([[_repo_json("kept")]], calls)
+
+    def urlopen(request, timeout=None):
+        headers.append(dict(request.headers))
+        return inner(request, timeout=timeout)
+
+    monkeypatch.setattr(sources.urllib.request, "urlopen", urlopen)
+    lines: list = []
+    assert [s.name for s in _expand(tmp_path, sources.OrgSource(org="acme"),
+                                    lines)] == ["kept"]
+    assert headers[0].get("Authorization") == "Bearer gho_secret"
+    # With a token the private repos are wanted too, so the filter comes off.
+    assert "type=public" not in calls[0]
+    assert not any("gho_secret" in line for line in lines)
+    cached = (tmp_path / "cache" / "org-acme.json").read_text(encoding="utf-8")
+    assert "gho_secret" not in cached
+
+
+# --- reading org entries out of the TOML file --------------------------------
+
+def test_an_org_entry_is_loaded_as_an_org_source(tmp_path: Path) -> None:
+    config = _write(tmp_path / "installer.toml", """
+[[source]]
+org = "AutomatedAlchemy"
+topic = "cli-tool-kit"
+exclude = ["alchemy-installer"]
+
+[[source]]
+name = "org/tools"
+path = "."
+""")
+    loaded = load_sources(config)
+    assert loaded[0] == sources.OrgSource(org="AutomatedAlchemy", topic="cli-tool-kit",
+                                          include=(), exclude=("alchemy-installer",))
+    assert loaded[1] == Source(name="org/tools", path=str(tmp_path))
+
+
+def test_the_topic_defaults_when_it_is_omitted(tmp_path: Path) -> None:
+    config = _write(tmp_path / "installer.toml", '[[source]]\norg = "acme"\n')
+    assert load_sources(config)[0].topic == sources.DEFAULT_ORG_TOPIC
+
+
+@pytest.mark.parametrize("body, complaint", [
+    ('org = "acme/sub"', "unusable org"),
+    ('org = "' + "a" * 40 + '"', "unusable org"),
+    ('org = ""', "unusable org"),
+    ('org = 7', "unusable org"),
+    ('org = "acme"\ntopic = "not a topic"', "not a usable topic"),
+    ('org = "acme"\nurl = "https://example.invalid/x.git"', "both org and url/path"),
+    ('org = "acme"\npath = "."', "both org and url/path"),
+    ('name = "acme"\norg = "acme"', "both org and url/path"),
+])
+def test_an_unusable_org_entry_is_reported_and_dropped(tmp_path: Path, body: str,
+                                                       complaint: str) -> None:
+    config = _write(tmp_path / "installer.toml", f"[[source]]\n{body}\n")
+    lines: list = []
+    assert load_sources(config, log=lines.append) == []
+    assert any(complaint in line for line in lines)
+
+
+def test_resolve_sources_ignores_an_unexpanded_org_entry(tmp_path: Path) -> None:
+    given = _repo(tmp_path / "given")
+    resolved = resolve_sources([sources.OrgSource(org="acme"),
+                                Source(name="lab", path=str(given))],
+                               tmp_path / "root")
+    assert resolved == [given]
+
+
+def test_run_installer_expands_an_org_entry_in_the_hook(tmp_path: Path, engine,
+                                                        api, monkeypatch) -> None:
+    config = _write(_repo(tmp_path / "org" / "tools") / "installer.toml", """
+[[source]]
+name = "org/tools"
+path = "."
+
+[[source]]
+org = "acme"
+""")
+    api([_repo_json("kept")])
+    git_calls: list = []
+    monkeypatch.setattr(sources.subprocess, "run", _fake_git(git_calls))
+    monkeypatch.chdir(tmp_path)
+    root = tmp_path / "root"
+    root.mkdir()
+    sources.run_installer(config, argv=["--root", str(root)])
+    roots = engine["discovery_roots"]
+    assert roots == [str(tmp_path / "org" / "tools")]       # nothing fetched yet
+    engine["pre_discovery"](False)
+    assert roots == [str(tmp_path / "org" / "tools"), str(root / "kept")]
+    assert any("clone" in call for call in git_calls)
+
+
+# --- local path pins on org-derived names ------------------------------------
+
+def _org_tree(tmp_path: Path, local_body: str) -> Path:
+    """installer.toml with only an org entry, plus a local file beside it."""
+    config = _write(tmp_path / "installer.toml", """
+[[source]]
+org = "AutomatedAlchemy"
+topic = "cli-tool-kit"
+exclude = ["alchemy-installer"]
+""")
+    _write(config.with_name("installer.local.toml"), local_body)
+    return config
+
+
+def test_a_local_pin_on_an_org_derived_name_is_used_without_cloning(
+        tmp_path: Path, api, monkeypatch) -> None:
+    """The repro: a local [[source]] naming a listed repo must pin its path.
+
+    The names do not exist when installer.local.toml is read, so the override
+    can only be matched after the listing — which is what regressed: both repos
+    were cloned over the checkouts that were already there.
+    """
+    root = tmp_path / "root"
+    pinned = _repo(root / "bloggen")
+    also = _repo(root / "lernclaude")
+    config = _org_tree(tmp_path, f"""
+root = "{root}"
+
+[[source]]
+name = "BlogGen"
+path = "{pinned}"
+
+[[source]]
+name = "lernclaude-fau"
+path = "{also}"
+""")
+    api([_repo_json("BlogGen"), _repo_json("lernclaude-fau"), _repo_json("manim-kit")])
+    git_calls: list = []
+    monkeypatch.setattr(sources.subprocess, "run", _fake_git(git_calls))
+
+    expanded = sources.expand_org_sources(load_sources(config),
+                                          cache_dir=tmp_path / "cache", root=root,
+                                          log=lambda *_: None)
+    assert [(s.name, s.path) for s in expanded] == [
+        ("BlogGen", str(pinned)), ("lernclaude-fau", str(also)), ("manim-kit", None)]
+
+    resolved = resolve_sources(expanded, root, log=lambda *_: None)
+    assert resolved[:2] == [pinned, also]
+    # The listing may be fetched; the pinned repos must not be cloned.
+    assert not any("clone" in call and "BlogGen" in " ".join(call)
+                   for call in git_calls)
+    assert not any("clone" in call and "lernclaude" in " ".join(call)
+                   for call in git_calls)
+
+
+def test_a_local_pin_whose_path_is_gone_still_clones(tmp_path: Path, api,
+                                                     monkeypatch) -> None:
+    root = tmp_path / "root"
+    root.mkdir()
+    config = _org_tree(tmp_path, f"""
+root = "{root}"
+
+[[source]]
+name = "BlogGen"
+path = "{tmp_path / 'never-checked-out'}"
+""")
+    api([_repo_json("BlogGen")])
+    git_calls: list = []
+    monkeypatch.setattr(sources.subprocess, "run", _fake_git(git_calls))
+    expanded = sources.expand_org_sources(load_sources(config),
+                                          cache_dir=tmp_path / "cache", root=root,
+                                          log=lambda *_: None)
+    assert expanded[0].url == "https://github.com/acme/BlogGen.git"
+    assert resolve_sources(expanded, root, log=lambda *_: None) == [root / "BlogGen"]
+    assert any("clone" in call for call in git_calls)
+
+
+def test_a_tracked_explicit_source_still_wins_over_the_listing(tmp_path: Path,
+                                                               api) -> None:
+    """The older rule is unchanged: a tracked [[source]] replaces the repo."""
+    fork = _repo(tmp_path / "fork")
+    config = _write(tmp_path / "installer.toml", f"""
+[[source]]
+name = "BlogGen"
+path = "{fork}"
+
+[[source]]
+org = "acme"
+""")
+    api([_repo_json("BlogGen"), _repo_json("manim-kit")])
+    expanded = sources.expand_org_sources(load_sources(config),
+                                          cache_dir=tmp_path / "cache",
+                                          log=lambda *_: None)
+    assert [(s.name, s.path, s.url) for s in expanded] == [
+        ("BlogGen", str(fork), None),
+        ("manim-kit", None, "https://github.com/acme/manim-kit.git")]