ghostpkg 0.5.0__tar.gz → 0.7.0__tar.gz

ghostpkg-0.7.0/2.0

@@ -0,0 +1 @@
+[REAL PIP] install requests flask

{ghostpkg-0.5.0 → ghostpkg-0.7.0}/CHANGELOG.md

@@ -6,6 +6,101 @@ this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.7.0] - 2026-09-01
+
+A correctness release. Several of these were **silent** failures -- the tool
+reported success having never performed the check, which for a security tool is
+worse than crashing. Found by a systematic audit rather than by users.
+
+### Fixed -- silent misses
+- **Every package whose name begins with `http` was dropped from
+  `requirements.txt`.** A filter meant to skip bare URLs matched names too, so
+  `httpx`, `httpcore`, `httplib2` and friends were never checked and the tool
+  still printed "all packages look fine". `httpx` is a top-100 PyPI project.
+- **`-r` and `-c` includes were ignored**, so a project that splits its
+  requirements across files was only half checked. Includes are now followed,
+  with cycle detection and a depth limit.
+
+### Fixed -- false blocks
+- **Direct references and private indexes.** `internal-lib @ git+https://...`
+  was read as the plain name `internal-lib`, which is absent from public PyPI,
+  so every internal package in a company using its own index was **blocked**.
+  Direct references, `--index-url`, editable installs, local paths and hashes
+  are now recognised and skipped.
+- **Any `.txt` file was parsed as requirements.** `README.txt` became the
+  package list `['Install', 'Run', 'numpy']` -- two confident blocks on prose.
+  Only `requirements*.txt`, `constraints*.txt` and `*.in` are accepted now, and
+  anything else is refused by name.
+- **`--deep` judged packages by install scripts they merely shipped.** Matching
+  `setup.py` by basename meant a package vendoring a dependency or including
+  packaging test fixtures was blocked on somebody else's code. Only the
+  archive's own top-level install script is read.
+
+### Fixed -- wrong exit codes and lost results
+- **A connection that failed mid-response escaped as a traceback and exited
+  `1`** -- the code meaning "a package does not exist". `urlopen` only wraps
+  failures that happen while connecting, so a stalled proxy or a reset
+  connection read as a confirmed detection. All lookup failures are now
+  `RegistryError`.
+- **One failed lookup discarded the entire scan**, including already-confirmed
+  blocks, and skipped the cache write so the inevitable retry re-issued every
+  request. Failures are now per-name, with a new `ERROR` verdict, and exit
+  code `2` -- an unchecked name is never a pass.
+- **`ghostpkg scan <directory>`** crashed with a traceback; it now explains
+  itself and exits `2`.
+
+### Fixed -- wrong answers
+- **npm cache keys collapsed case-distinct packages.** `JSONStream` and
+  `jsonstream` are two different real packages; lowercasing both into one key
+  served one package's facts for the other, including a wrong `exists`. Keys
+  are now normalised per ecosystem -- PEP 503 for PyPI, exact for npm.
+
+### Changed
+- **npm lookups now request gzip.** `@types/node` was 10.6 MB per lookup and is
+  now 1.35 MB; `react` went from 6.6 MB to 1.30 MB. Responses are also size-capped.
+- **Repeated names are looked up once.** Manifests repeat names, and following
+  includes makes that more likely.
+- `pyproject.toml` now reads PEP 735 `[dependency-groups]`, and Poetry group
+  dependencies are found on Python 3.9/3.10 as well as 3.11+.
+- `package.json` now reads `peerDependencies`.
+- The cache takes a lock around its counters and entry map, and its docstring
+  no longer claims threads only read from it -- they do not.
+
+## [0.6.0] - 2026-09-01
+
+### Fixed
+- **Typo detection missed parked lookalikes, because age was the wrong gate.**
+  The check only ran on packages published within the last year, on the
+  reasoning that "an old lookalike is just a package with a similar name". That
+  reasoning was wrong. `expresss` has sat on npm since **2016** with one
+  release, no repository link, and roughly **2,500 downloads a month** arriving
+  purely from other people's typos. Ten years old, and waved straight through.
+
+### Changed
+- The typo check now also runs on packages that look **abandoned**: two or fewer
+  releases *and* no repository link, at any age.
+
+  The pairing is not a guess. Measured against 120 real packages that sit within
+  the typo budget of a popular name:
+
+  | Rule | Wrong about |
+  |---|---|
+  | Two or fewer releases, alone | 10.0% |
+  | No repository link, alone | 5.8% |
+  | **Both together** | **0.0%** |
+
+  Either condition alone is too loose because sibling packages in a family sit
+  naturally close together — `dagster-k8s` is two edits from `dagster-aws`, and
+  `pulumi-tls` from `pulumi-aws`. Those are maintained, so they carry releases
+  and a repository, and requiring both conditions leaves them alone.
+
+- Warning text now says which condition fired, rather than calling a
+  ten-year-old package "recently published".
+
+### Note
+A defensively parked name still passes, which is correct: npm holds `lodahs`
+itself and points it at `npm/security-holder`, so it has a repository link.
+
 ## [0.5.0] - 2026-09-01
 
 ### Added
@@ -149,7 +244,9 @@ First release.
 - No corpus of hallucinated package names is shipped, following the decision of
   the USENIX'25 authors not to publish theirs.
 
-[Unreleased]: https://github.com/M1rwana12/ghostpkg/compare/v0.5.0...HEAD
+[Unreleased]: https://github.com/M1rwana12/ghostpkg/compare/v0.7.0...HEAD
+[0.7.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.7.0
+[0.6.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.6.0
 [0.5.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.5.0
 [0.4.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.4.0
 [0.3.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.3.0

{ghostpkg-0.5.0 → ghostpkg-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ghostpkg
-Version: 0.5.0
+Version: 0.7.0
 Summary: Catch package names that do not exist before you install them
 Project-URL: Homepage, https://github.com/m1rwana12/ghostpkg
 Project-URL: Issues, https://github.com/m1rwana12/ghostpkg/issues
@@ -238,7 +238,7 @@ $ ghostpkg check somepkgthatisnotreal9911 --json
 | First published < 1 year ago | 🟡 Warning | Weaker version of the same signal. |
 | Only one release | 🟡 Warning | Squats are usually published once and abandoned. |
 | No repository or homepage link | 🟡 Warning | Real projects almost always link to source. |
-| 1–2 edits from a popular name, **and** recently published | 🟡 Warning | Classic typosquat shape. A swap of adjacent characters counts as one edit, because `recat`/`react` is what squatters actually publish. Age matters: an old lookalike is just a package with a similar name. |
+| 1–2 edits from a popular name, **and** the package is either recent **or** abandoned (≤2 releases and no repository link) | 🟡 Warning | Classic typosquat shape. A swap of adjacent characters counts as one edit, because `recat`/`react` is what squatters actually publish. Age alone was the wrong gate: `expresss` has sat on npm since 2016 with one release and ~2,500 typo-driven downloads a month. |
 
 Warnings are advisory by default. Nothing but non-existence blocks unless you pass
 `--strict`.

{ghostpkg-0.5.0 → ghostpkg-0.7.0}/README.en.md

@@ -191,7 +191,7 @@ $ ghostpkg check somepkgthatisnotreal9911 --json
 | First published < 1 year ago | 🟡 Warning | Weaker version of the same signal. |
 | Only one release | 🟡 Warning | Squats are usually published once and abandoned. |
 | No repository or homepage link | 🟡 Warning | Real projects almost always link to source. |
-| 1–2 edits from a popular name, **and** recently published | 🟡 Warning | Classic typosquat shape. A swap of adjacent characters counts as one edit, because `recat`/`react` is what squatters actually publish. Age matters: an old lookalike is just a package with a similar name. |
+| 1–2 edits from a popular name, **and** the package is either recent **or** abandoned (≤2 releases and no repository link) | 🟡 Warning | Classic typosquat shape. A swap of adjacent characters counts as one edit, because `recat`/`react` is what squatters actually publish. Age alone was the wrong gate: `expresss` has sat on npm since 2016 with one release and ~2,500 typo-driven downloads a month. |
 
 Warnings are advisory by default. Nothing but non-existence blocks unless you pass
 `--strict`.

{ghostpkg-0.5.0 → ghostpkg-0.7.0}/README.md

@@ -118,7 +118,7 @@ ghostpkg clear-cache
 | Опубліковано днями тому | 🟡 Попередження. Зловмисники реєструють швидко — але й чесні автори теж. |
 | Лише один реліз | 🟡 Попередження. |
 | Немає посилання на репозиторій | 🟡 Попередження. |
-| За один-два символи від популярної назви **і** свіжий | 🟡 Попередження. Схоже на typosquat. |
+| За один-два символи від популярної назви, **і** пакет свіжий **або** занедбаний (≤2 релізи й немає репозиторію) | 🟡 Попередження. Схоже на typosquat. |
 
 ---
 

{ghostpkg-0.5.0 → ghostpkg-0.7.0}/ghostpkg/__init__.py

@@ -1,6 +1,6 @@
 """ghostpkg -- catch package names that do not exist before you install them."""
 
-__version__ = "0.5.0"
+__version__ = "0.7.0"
 
 from .assess import Finding, Verdict, assess
 from .registries import PackageFacts, RegistryError, fetch

{ghostpkg-0.5.0 → ghostpkg-0.7.0}/ghostpkg/assess.py

@@ -29,6 +29,10 @@ class Verdict(str, Enum):
     OK = "OK"
     WARN = "WARN"
     BLOCK = "BLOCK"
+    #: The registry could not be reached for this name. Never a pass: a
+    #: security check that silently succeeds when it could not run is worse
+    #: than no check at all.
+    ERROR = "ERROR"
 
 
 @dataclass
@@ -43,6 +47,10 @@ class Finding:
     def is_blocked(self) -> bool:
         return self.verdict is Verdict.BLOCK
 
+    @property
+    def is_error(self) -> bool:
+        return self.verdict is Verdict.ERROR
+
 
 def edit_distance(left: str, right: str, cutoff: int = 3) -> int:
     """Damerau-Levenshtein distance, abandoning early once it exceeds `cutoff`.
@@ -173,13 +181,27 @@ def assess(
 
     is_young = facts.age_days is not None and facts.age_days < NEW_DAYS
 
-    if is_young:
+    # A parked lookalike is not necessarily new. `expresss` has sat on npm since
+    # 2016 with one release, no repository link, and roughly 2,500 downloads a
+    # month arriving purely from other people's typos. Age was the wrong gate.
+    #
+    # Abandonment is the right one, but only as a pair. Measured against 120 real
+    # packages that sit within the typo budget of a popular name, "few releases"
+    # alone was wrong 10% of the time and "no repository link" alone 5.8%, while
+    # requiring both was wrong 0% of the time. That matters because sibling
+    # packages in a family are naturally close together -- dagster-k8s is two
+    # edits from dagster-aws, pulumi-tls from pulumi-aws -- and they are
+    # maintained, so they carry releases and a repository.
+    looks_abandoned = facts.release_count <= 2 and not facts.has_repo_url
+
+    if is_young or looks_abandoned:
         neighbour = nearest_popular(facts.name, facts.ecosystem)
         if neighbour is not None:
             popular_name, distance = neighbour
+            character = "character" if distance == 1 else "characters"
+            context = "recently published" if is_young else "one release, no repository"
             reasons.append(
-                f"{distance} character{'s' if distance > 1 else ''} away from "
-                f"'{popular_name}', and recently published"
+                f"{distance} {character} away from '{popular_name}', and {context}"
             )
 
     if is_young and facts.release_count <= 1:

{ghostpkg-0.5.0 → ghostpkg-0.7.0}/ghostpkg/cache.py

@@ -17,8 +17,10 @@ from __future__ import annotations
 
 import json
 import os
+import re
 import sys
 import tempfile
+import threading
 import time
 from dataclasses import asdict
 from pathlib import Path
@@ -61,7 +63,15 @@ def ttl_for(facts: PackageFacts) -> int:
 
 
 class Cache:
-    """Read once, write once. Threads only read, so no locking is needed."""
+    """Loaded once, written once, read and updated from worker threads between.
+
+    `put` is called from the scan's thread pool, so this is not lock-free by
+    virtue of nobody mutating it -- an earlier version of this docstring
+    claimed that and was wrong. It is safe because `dict.__setitem__` is
+    atomic under the GIL and `save`/`_prune` only run after the pool has shut
+    down. The lock exists so that stops being load-bearing on free-threaded
+    builds, where that guarantee does not hold.
+    """
 
     def __init__(self, directory: Path | None = None, enabled: bool = True) -> None:
         self.enabled = enabled
@@ -70,6 +80,7 @@ class Cache:
         self._dirty = False
         self.hits = 0
         self.misses = 0
+        self._lock = threading.Lock()
         if self.enabled:
             self._load()
 
@@ -90,44 +101,62 @@ class Cache:
 
     @staticmethod
     def _key(ecosystem: str, name: str) -> str:
-        return f"{ecosystem}:{name.lower()}"
+        """Normalise per ecosystem, because they do not agree.
+
+        PyPI names are case-insensitive and treat `-`, `_` and `.` as the same
+        separator (PEP 503), so `zope.interface` and `Zope_Interface` are one
+        package. npm names are case-sensitive, and `JSONStream` and `jsonstream`
+        are two different real packages -- lowercasing both into one key served
+        one package's facts for the other, including a wrong `exists`.
+        """
+        if ecosystem == "pypi":
+            return f"pypi:{re.sub(r'[-_.]+', '-', name).lower()}"
+        return f"{ecosystem}:{name}"
 
     def get(self, ecosystem: str, name: str) -> PackageFacts | None:
         if not self.enabled:
             return None
-        entry = self._entries.get(self._key(ecosystem, name))
+        with self._lock:
+            entry = self._entries.get(self._key(ecosystem, name))
         if not isinstance(entry, dict):
-            self.misses += 1
+            self._miss()
             return None
         stored_at = entry.get("at")
         ttl = entry.get("ttl")
         facts = entry.get("facts")
         if not isinstance(stored_at, (int, float)) or not isinstance(ttl, (int, float)):
-            self.misses += 1
+            self._miss()
             return None
         if time.time() - stored_at > ttl:
-            self.misses += 1
+            self._miss()
             return None
         try:
             revived = PackageFacts(**facts)
         except (TypeError, ValueError):
-            self.misses += 1
+            self._miss()
             return None
-        self.hits += 1
+        with self._lock:
+            self.hits += 1
         # keep the caller's spelling of the name rather than the cached one
         if revived.name != name:
             revived = PackageFacts(**{**asdict(revived), "name": name})
         return revived
 
+    def _miss(self) -> None:
+        with self._lock:
+            self.misses += 1
+
     def put(self, facts: PackageFacts) -> None:
         if not self.enabled:
             return
-        self._entries[self._key(facts.ecosystem, facts.name)] = {
+        entry = {
             "at": time.time(),
             "ttl": ttl_for(facts),
             "facts": asdict(facts),
         }
-        self._dirty = True
+        with self._lock:
+            self._entries[self._key(facts.ecosystem, facts.name)] = entry
+            self._dirty = True
 
     def _prune(self) -> None:
         now = time.time()

{ghostpkg-0.5.0 → ghostpkg-0.7.0}/ghostpkg/cli.py

@@ -51,6 +51,7 @@ class Palette:
 
 MARKS = {
     Verdict.BLOCK: ("BLOCKED", "red"),
+    Verdict.ERROR: ("ERROR", "red"),
     Verdict.WARN: ("WARNING", "yellow"),
     Verdict.OK: ("ok", "green"),
 }
@@ -80,14 +81,18 @@ def render(findings: list[Finding], palette: Palette, quiet: bool) -> None:
 def summarise(findings: list[Finding], palette: Palette) -> None:
     blocked = [f for f in findings if f.verdict is Verdict.BLOCK]
     warned = [f for f in findings if f.verdict is Verdict.WARN]
+    errored = [f for f in findings if f.verdict is Verdict.ERROR]
 
     print()
     if blocked:
         names = ", ".join(f.name for f in blocked)
         print(palette.red(f"  {len(blocked)} blocked: {names}"))
+    if errored:
+        names = ", ".join(f.name for f in errored)
+        print(palette.red(f"  {len(errored)} could not be checked: {names}"))
     if warned:
         print(palette.yellow(f"  {len(warned)} to review by hand"))
-    if not blocked and not warned:
+    if not blocked and not warned and not errored:
         print(palette.green(f"  all {len(findings)} packages look fine"))
 
 
@@ -99,11 +104,24 @@ def evaluate(
     deep: bool = False,
 ) -> list[Finding]:
     def one(name: str) -> Finding:
-        facts = cache.get(ecosystem, name) if cache else None
-        if facts is None:
-            facts = fetch(name, ecosystem)
-            if cache:
-                cache.put(facts)
+        # A failure on one name must not discard the whole scan. It used to:
+        # pool.map re-raised the first exception while iterating, throwing away
+        # every confirmed BLOCK alongside it and skipping the cache write, so
+        # the inevitable retry re-issued every lookup and made a rate-limit
+        # response self-amplifying.
+        try:
+            facts = cache.get(ecosystem, name) if cache else None
+            if facts is None:
+                facts = fetch(name, ecosystem)
+                if cache:
+                    cache.put(facts)
+        except RegistryError as exc:
+            return Finding(
+                name=name,
+                ecosystem=ecosystem,
+                verdict=Verdict.ERROR,
+                reasons=[f"could not check: {exc}"],
+            )
 
         signals = None
         # Only young packages are worth the download: a registered slopsquat is
@@ -124,11 +142,14 @@ def evaluate(
 
         return assess(facts, strict=strict, signals=signals)
 
+    # Look each distinct name up once. A manifest can repeat a name, and
+    # following -r includes makes that more likely.
+    unique = list(dict.fromkeys(names))
     with concurrent.futures.ThreadPoolExecutor(max_workers=MAX_WORKERS) as pool:
-        findings = list(pool.map(one, names))
+        results = dict(zip(unique, pool.map(one, unique)))
     if cache:
         cache.save()
-    return findings
+    return [results[name] for name in names]
 
 
 def build_parser() -> argparse.ArgumentParser:
@@ -186,12 +207,18 @@ def main(argv: list[str] | None = None) -> int:
         if not args.path.exists():
             print(f"ghostpkg: no such file: {args.path}", file=sys.stderr)
             return EXIT_ERROR
+        if args.path.is_dir():
+            print(
+                f"ghostpkg: {args.path} is a directory; pass a manifest file",
+                file=sys.stderr,
+            )
+            return EXIT_ERROR
         try:
             names, ecosystem = load_manifest(args.path)
         except UnsupportedManifest as exc:
             print(f"ghostpkg: {exc}", file=sys.stderr)
             return EXIT_ERROR
-        except (json.JSONDecodeError, UnicodeDecodeError, ValueError) as exc:
+        except (json.JSONDecodeError, UnicodeDecodeError, ValueError, OSError) as exc:
             print(f"ghostpkg: could not parse {args.path}: {exc}", file=sys.stderr)
             return EXIT_ERROR
         if not names:
@@ -201,11 +228,7 @@ def main(argv: list[str] | None = None) -> int:
         names, ecosystem = args.names, args.ecosystem
 
     cache = Cache(enabled=not args.no_cache)
-    try:
-        findings = evaluate(names, ecosystem, args.strict, cache, args.deep)
-    except RegistryError as exc:
-        print(f"ghostpkg: {exc}", file=sys.stderr)
-        return EXIT_ERROR
+    findings = evaluate(names, ecosystem, args.strict, cache, args.deep)
 
     if args.json:
         print(
@@ -227,7 +250,12 @@ def main(argv: list[str] | None = None) -> int:
         render(findings, palette, args.quiet)
         summarise(findings, palette)
 
-    return EXIT_BLOCKED if any(f.is_blocked for f in findings) else EXIT_OK
+    if any(f.is_blocked for f in findings):
+        return EXIT_BLOCKED
+    # An unchecked name is not a pass.
+    if any(f.is_error for f in findings):
+        return EXIT_ERROR
+    return EXIT_OK
 
 
 if __name__ == "__main__":

{ghostpkg-0.5.0 → ghostpkg-0.7.0}/ghostpkg/inspection.py

@@ -146,7 +146,20 @@ def _read_member(handle, member, size: int) -> str | None:
 
 
 def _interesting(path: str, ecosystem: str) -> bool:
-    name = path.rsplit("/", 1)[-1]
+    """Only the archive's own install script, never a nested one.
+
+    Matching on the basename alone meant a package that ships packaging test
+    fixtures or vendors a dependency was judged on somebody else's
+    `setup.py` -- and since install signals block a young package, that was
+    the one false-positive path in the blocking logic.
+
+    Both ecosystems put the real one exactly one directory down:
+    `<name>-<version>/setup.py` for an sdist, `package/package.json` for npm.
+    """
+    parts = path.strip("/").split("/")
+    if len(parts) != 2:
+        return False
+    name = parts[1]
     if ecosystem == "npm":
         return name == "package.json"
     return name in PY_INSTALL_FILES