ghostpkg 0.1.0__tar.gz → 0.3.0__tar.gz

ghostpkg-0.3.0/CHANGELOG.md

@@ -0,0 +1,93 @@
+# Changelog
+
+All notable changes to this project are documented here.
+Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
+this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.3.0] - 2026-09-01
+
+### Fixed
+- **npm typo detection did not work at all.** Lookalike names were compared
+  against the 2,000 most-downloaded *PyPI* projects regardless of ecosystem, so
+  `expresss` was never flagged as a typo of `express` -- `express` was not in
+  the list being compared against.
+
+### Added
+- A list of the 2,000 most-downloaded npm packages, built from the registry's
+  own download-count API. `nearest_popular()` now picks the list matching the
+  ecosystem.
+- Scoped npm names are compared on the part after the slash, since that is what
+  a squat targets: `@evil/expresss` is flagged, `@types/node` and `@babel/core`
+  are not.
+
+### Changed
+- Typo distance is now Damerau-Levenshtein: swapping two adjacent characters
+  counts as one edit rather than two. Transposition is the commonest typosquat
+  shape, and under plain Levenshtein `recat`, `lodahs` and `webpakc` all scored
+  two edits, which put them outside the budget for names that short. All three
+  are caught now, and the false-positive rate across both 2,000-name lists is
+  still zero -- there are tests asserting exactly that.
+- Names shorter than five characters are no longer compared at all. Below that
+  the name space is too dense for edit distance to mean anything: `core` sits
+  one edit from `cors`.
+
+## [0.2.0] - 2026-09-01
+
+### Fixed
+- **`scan` reported nonsense for `pyproject.toml`.** Every file that was not
+  `package.json` was handed to the requirements.txt parser, so TOML keys were
+  read as package names: `build-backend` came back "does not exist on pypi",
+  and `version` came back "ok" because a package of that name exists. For a
+  security tool, confidently reporting nonsense is the worst failure mode.
+
+### Added
+- `pyproject.toml` support in `scan`: PEP 621 `project.dependencies` and
+  `project.optional-dependencies`, plus Poetry's `tool.poetry` groups.
+  `[build-system] requires` is excluded, and Poetry's `python` constraint is
+  not treated as a package.
+- Manifests are now detected explicitly. An unrecognised file is **refused with
+  an error rather than guessed at** — guessing was the root cause of the bug
+  above.
+- A narrow TOML fallback parser for Python 3.9 and 3.10, which have no
+  `tomllib`. Adding a TOML dependency would contradict the zero-dependency rule,
+  and the fallback is tested by forcing the import to fail rather than by
+  trusting the interpreter the suite runs on.
+
+### Changed
+- `--version` now reads `__version__` instead of a hardcoded string.
+
+## [0.1.0] - 2026-09-01
+
+First release.
+
+### Added
+- `ghostpkg check` — check one or more package names against PyPI or npm.
+- `ghostpkg scan` — check every dependency in `requirements.txt` or `package.json`.
+- Verdicts: `BLOCK` for names absent from the registry, `WARN` for weak signals
+  (recent publication, single release, missing repository link, small edit
+  distance to a popular name), `OK` otherwise.
+- `--strict` to promote warnings to blocks, `--json` for machine-readable output,
+  `-q/--quiet` to hide passing packages.
+- Exit codes: `0` nothing blocked, `1` something blocked, `2` usage or registry
+  error.
+- Bundled list of the 2,000 most-downloaded PyPI projects, used only for typo
+  distance.
+
+### Notes on the design
+- The default profile blocks on **non-existence only**. An earlier version scored
+  packages and blocked anything suspicious; measured against the live feed of
+  newly published PyPI packages, it flagged 100% of legitimate same-day
+  publications. Age cannot separate a malicious registration from an honest new
+  release, so softer signals are warnings.
+- The typo-distance budget scales with name length. A flat budget flagged `flask`,
+  `click` and `black` as typos of one another.
+- Zero runtime dependencies, by policy.
+- No corpus of hallucinated package names is shipped, following the decision of
+  the USENIX'25 authors not to publish theirs.
+
+[Unreleased]: https://github.com/M1rwana12/ghostpkg/compare/v0.3.0...HEAD
+[0.3.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.3.0
+[0.2.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.2.0
+[0.1.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.1.0

{ghostpkg-0.1.0 → ghostpkg-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ghostpkg
-Version: 0.1.0
+Version: 0.3.0
 Summary: Catch package names that do not exist before you install them
 Project-URL: Homepage, https://github.com/m1rwana12/ghostpkg
 Project-URL: Issues, https://github.com/m1rwana12/ghostpkg/issues
@@ -59,8 +59,8 @@ Language models invent library names. Attackers register those names in advance.
 **English** · [Українська](https://github.com/M1rwana12/ghostpkg/blob/main/README.md)
 
 [![CI](https://github.com/M1rwana12/ghostpkg/actions/workflows/ci.yml/badge.svg)](https://github.com/M1rwana12/ghostpkg/actions/workflows/ci.yml)
-[![PyPI](https://img.shields.io/pypi/v/ghostpkg.svg?color=A78BFA)](https://pypi.org/project/ghostpkg/)
-[![Python](https://img.shields.io/pypi/pyversions/ghostpkg.svg?color=A78BFA)](https://pypi.org/project/ghostpkg/)
+[![PyPI](https://img.shields.io/pypi/v/ghostpkg?color=A78BFA&label=pypi)](https://pypi.org/project/ghostpkg/)
+[![Python](https://img.shields.io/pypi/pyversions/ghostpkg?color=A78BFA&label=python)](https://pypi.org/project/ghostpkg/)
 [![Dependencies](https://img.shields.io/badge/dependencies-0-3FB950.svg)](https://github.com/M1rwana12/ghostpkg/blob/main/pyproject.toml)
 [![License](https://img.shields.io/badge/license-MIT-8B949E.svg)](https://github.com/M1rwana12/ghostpkg/blob/main/LICENSE)
 
@@ -158,12 +158,19 @@ ghostpkg check react-router-dom-utils -e npm
 
 ```bash
 ghostpkg scan requirements.txt
+ghostpkg scan pyproject.toml
 ghostpkg scan package.json
 ```
 
-`requirements.txt` parsing skips comments, flags, VCS URLs and direct links, and
-takes the bare project name off each requirement line. `package.json` reads
-`dependencies`, `devDependencies` and `optionalDependencies`.
+| Manifest | What is read |
+|---|---|
+| `requirements*.txt` | Every requirement line. Comments, flags, VCS URLs and direct links are skipped. |
+| `pyproject.toml` | PEP 621 `project.dependencies` and `project.optional-dependencies`, plus Poetry's `tool.poetry` groups. `[build-system] requires` is not included, and the `python` constraint is not treated as a package. |
+| `package.json` | `dependencies`, `devDependencies`, `optionalDependencies`. |
+
+Anything else is **refused with an error rather than guessed at**. Guessing is
+what made an earlier version read `pyproject.toml` with the requirements parser
+and report TOML keys as package names.
 
 ### Options
 
@@ -210,7 +217,7 @@ $ ghostpkg check somepkgthatisnotreal9911 --json
 | First published < 1 year ago | 🟡 Warning | Weaker version of the same signal. |
 | Only one release | 🟡 Warning | Squats are usually published once and abandoned. |
 | No repository or homepage link | 🟡 Warning | Real projects almost always link to source. |
-| 1–2 characters from a popular name, **and** recently published | 🟡 Warning | Classic typosquat shape. Age matters: an old lookalike is just a package with a similar name. |
+| 1–2 edits from a popular name, **and** recently published | 🟡 Warning | Classic typosquat shape. A swap of adjacent characters counts as one edit, because `recat`/`react` is what squatters actually publish. Age matters: an old lookalike is just a package with a similar name. |
 
 Warnings are advisory by default. Nothing but non-existence blocks unless you pass
 `--strict`.
@@ -346,10 +353,10 @@ earns its keep. Use both.
 > that stand between you and it, and they are advisory. Improving this is the main
 > open problem — see [issues](https://github.com/M1rwana12/ghostpkg/issues).
 
-- Typo detection compares against the 2,000 most-downloaded PyPI projects, so a squat
-  on a less popular package won't be flagged as a lookalike.
-- npm scoped packages (`@scope/name`) are checked, but the popular-name list is
-  PyPI-derived, so npm typosquat detection is weaker.
+- Typo detection compares against the 2,000 most-downloaded projects in each
+  ecosystem, so a squat on a less popular package won't be flagged as a lookalike.
+- Names shorter than five characters are not compared at all: below that the name
+  space is too dense for edit distance to mean anything.
 - Every check is a live registry request. There is no caching yet.
 - Registry outages surface as exit code `2` rather than a silent pass — deliberately,
   but it does mean a flaky network fails your build.

{ghostpkg-0.1.0 → ghostpkg-0.3.0}/README.en.md

@@ -12,8 +12,8 @@ Language models invent library names. Attackers register those names in advance.
 **English** · [Українська](https://github.com/M1rwana12/ghostpkg/blob/main/README.md)
 
 [![CI](https://github.com/M1rwana12/ghostpkg/actions/workflows/ci.yml/badge.svg)](https://github.com/M1rwana12/ghostpkg/actions/workflows/ci.yml)
-[![PyPI](https://img.shields.io/pypi/v/ghostpkg.svg?color=A78BFA)](https://pypi.org/project/ghostpkg/)
-[![Python](https://img.shields.io/pypi/pyversions/ghostpkg.svg?color=A78BFA)](https://pypi.org/project/ghostpkg/)
+[![PyPI](https://img.shields.io/pypi/v/ghostpkg?color=A78BFA&label=pypi)](https://pypi.org/project/ghostpkg/)
+[![Python](https://img.shields.io/pypi/pyversions/ghostpkg?color=A78BFA&label=python)](https://pypi.org/project/ghostpkg/)
 [![Dependencies](https://img.shields.io/badge/dependencies-0-3FB950.svg)](https://github.com/M1rwana12/ghostpkg/blob/main/pyproject.toml)
 [![License](https://img.shields.io/badge/license-MIT-8B949E.svg)](https://github.com/M1rwana12/ghostpkg/blob/main/LICENSE)
 
@@ -111,12 +111,19 @@ ghostpkg check react-router-dom-utils -e npm
 
 ```bash
 ghostpkg scan requirements.txt
+ghostpkg scan pyproject.toml
 ghostpkg scan package.json
 ```
 
-`requirements.txt` parsing skips comments, flags, VCS URLs and direct links, and
-takes the bare project name off each requirement line. `package.json` reads
-`dependencies`, `devDependencies` and `optionalDependencies`.
+| Manifest | What is read |
+|---|---|
+| `requirements*.txt` | Every requirement line. Comments, flags, VCS URLs and direct links are skipped. |
+| `pyproject.toml` | PEP 621 `project.dependencies` and `project.optional-dependencies`, plus Poetry's `tool.poetry` groups. `[build-system] requires` is not included, and the `python` constraint is not treated as a package. |
+| `package.json` | `dependencies`, `devDependencies`, `optionalDependencies`. |
+
+Anything else is **refused with an error rather than guessed at**. Guessing is
+what made an earlier version read `pyproject.toml` with the requirements parser
+and report TOML keys as package names.
 
 ### Options
 
@@ -163,7 +170,7 @@ $ ghostpkg check somepkgthatisnotreal9911 --json
 | First published < 1 year ago | 🟡 Warning | Weaker version of the same signal. |
 | Only one release | 🟡 Warning | Squats are usually published once and abandoned. |
 | No repository or homepage link | 🟡 Warning | Real projects almost always link to source. |
-| 1–2 characters from a popular name, **and** recently published | 🟡 Warning | Classic typosquat shape. Age matters: an old lookalike is just a package with a similar name. |
+| 1–2 edits from a popular name, **and** recently published | 🟡 Warning | Classic typosquat shape. A swap of adjacent characters counts as one edit, because `recat`/`react` is what squatters actually publish. Age matters: an old lookalike is just a package with a similar name. |
 
 Warnings are advisory by default. Nothing but non-existence blocks unless you pass
 `--strict`.
@@ -299,10 +306,10 @@ earns its keep. Use both.
 > that stand between you and it, and they are advisory. Improving this is the main
 > open problem — see [issues](https://github.com/M1rwana12/ghostpkg/issues).
 
-- Typo detection compares against the 2,000 most-downloaded PyPI projects, so a squat
-  on a less popular package won't be flagged as a lookalike.
-- npm scoped packages (`@scope/name`) are checked, but the popular-name list is
-  PyPI-derived, so npm typosquat detection is weaker.
+- Typo detection compares against the 2,000 most-downloaded projects in each
+  ecosystem, so a squat on a less popular package won't be flagged as a lookalike.
+- Names shorter than five characters are not compared at all: below that the name
+  space is too dense for edit distance to mean anything.
 - Every check is a live registry request. There is no caching yet.
 - Registry outages surface as exit code `2` rather than a silent pass — deliberately,
   but it does mean a flaky network fails your build.

{ghostpkg-0.1.0 → ghostpkg-0.3.0}/README.md

@@ -12,8 +12,8 @@
 [English](README.en.md) · **Українська**
 
 [![CI](https://github.com/M1rwana12/ghostpkg/actions/workflows/ci.yml/badge.svg)](https://github.com/M1rwana12/ghostpkg/actions/workflows/ci.yml)
-[![PyPI](https://img.shields.io/pypi/v/ghostpkg.svg?color=A78BFA)](https://pypi.org/project/ghostpkg/)
-[![Python](https://img.shields.io/pypi/pyversions/ghostpkg.svg?color=A78BFA)](https://pypi.org/project/ghostpkg/)
+[![PyPI](https://img.shields.io/pypi/v/ghostpkg?color=A78BFA&label=pypi)](https://pypi.org/project/ghostpkg/)
+[![Python](https://img.shields.io/pypi/pyversions/ghostpkg?color=A78BFA&label=python)](https://pypi.org/project/ghostpkg/)
 [![Залежності](https://img.shields.io/badge/%D0%B7%D0%B0%D0%BB%D0%B5%D0%B6%D0%BD%D0%BE%D1%81%D1%82%D0%B5%D0%B9-0-3FB950.svg)](pyproject.toml)
 [![Ліцензія](https://img.shields.io/badge/%D0%BB%D1%96%D1%86%D0%B5%D0%BD%D0%B7%D1%96%D1%8F-MIT-8B949E.svg)](LICENSE)
 
@@ -81,6 +81,7 @@ ghostpkg check react-router-dom-utils -e npm
 
 # перевірити всі залежності з маніфесту
 ghostpkg scan requirements.txt
+ghostpkg scan pyproject.toml
 ghostpkg scan package.json
 
 # машинозчитуваний вивід
@@ -94,6 +95,9 @@ ghostpkg check somepkg --json
 | `--json` | Вивід у JSON для скриптів і CI |
 | `-q`, `--quiet` | Ховає пакети, які пройшли перевірку |
 
+`scan` розпізнає `requirements*.txt`, `pyproject.toml` (PEP 621 і Poetry)
+та `package.json`. Невідомий формат він **відхиляє з помилкою, а не вгадує**.
+
 ---
 
 ## Що саме перевіряється
@@ -159,10 +163,10 @@ $ ghostpkg check react-router-dom-utils -e npm
 > попередження, і вони дорадчі. Це головна відкрита проблема —
 > див. [issues](https://github.com/M1rwana12/ghostpkg/issues).
 
-- Виявлення опечаток порівнює з 2 000 найпопулярніших проєктів PyPI, тому підробка
-  під менш популярний пакет як схожа назва не позначиться.
-- Пакети npm з областю (`@scope/name`) перевіряються, але список популярних назв
-  побудований на PyPI — тож для npm виявлення опечаток слабше.
+- Виявлення опечаток порівнює з 2 000 найпопулярніших проєктів кожної екосистеми,
+  тому підробка під менш популярний пакет як схожа назва не позначиться.
+- Імена коротші за 5 символів не порівнюються: там простір назв надто щільний,
+  щоб відстань редагування щось означала.
 - Кожна перевірка — це живий запит до реєстру. Кешування ще немає.
 
 ---

{ghostpkg-0.1.0 → ghostpkg-0.3.0}/ghostpkg/__init__.py

@@ -1,6 +1,6 @@
 """ghostpkg -- catch package names that do not exist before you install them."""
 
-__version__ = "0.1.0"
+__version__ = "0.3.0"
 
 from .assess import Finding, Verdict, assess
 from .registries import PackageFacts, RegistryError, fetch

{ghostpkg-0.1.0 → ghostpkg-0.3.0}/ghostpkg/assess.py

@@ -18,7 +18,7 @@ from __future__ import annotations
 from dataclasses import dataclass, field
 from enum import Enum
 
-from .data import TOP_PYPI
+from .data import TOP_NPM, TOP_PYPI
 from .registries import PackageFacts
 
 YOUNG_DAYS = 90
@@ -45,25 +45,44 @@ class Finding:
 
 
 def edit_distance(left: str, right: str, cutoff: int = 3) -> int:
-    """Levenshtein distance, abandoning early once it exceeds `cutoff`."""
+    """Damerau-Levenshtein distance, abandoning early once it exceeds `cutoff`.
+
+    Swapping two adjacent characters counts as one edit, not two. That matters
+    here more than it looks: transposition is the commonest typosquat shape --
+    `recat` for `react`, `lodahs` for `lodash`, `webpakc` for `webpack`. Plain
+    Levenshtein scores all three as two edits, which put them outside the budget
+    for names of that length and let every one of them through.
+    """
     if left == right:
         return 0
     if abs(len(left) - len(right)) > cutoff:
         return cutoff + 1
 
+    before_previous: list[int] = []
     previous = list(range(len(right) + 1))
     for i, a in enumerate(left, 1):
         current = [i]
         for j, b in enumerate(right, 1):
-            current.append(
-                min(previous[j] + 1, current[j - 1] + 1, previous[j - 1] + (a != b))
-            )
+            cost = min(previous[j] + 1, current[j - 1] + 1, previous[j - 1] + (a != b))
+            if (
+                i > 1
+                and j > 1
+                and a == right[j - 2]
+                and left[i - 2] == b
+            ):
+                cost = min(cost, before_previous[j - 2] + 1)
+            current.append(cost)
         if min(current) > cutoff:
             return cutoff + 1
-        previous = current
+        before_previous, previous = previous, current
     return previous[-1]
 
 
+POPULAR: dict[str, frozenset[str]] = {"pypi": TOP_PYPI, "npm": TOP_NPM}
+
+MIN_COMPARABLE_LENGTH = 5
+
+
 def _typo_budget(name: str) -> int:
     """How many edits still count as a plausible typo of a popular name.
 
@@ -74,18 +93,40 @@ def _typo_budget(name: str) -> int:
     return 2 if len(name) >= 10 else 1
 
 
-def nearest_popular(name: str, popular: frozenset[str] = TOP_PYPI) -> tuple[str, int] | None:
+def _comparable(name: str, ecosystem: str) -> str:
+    """The part of a name worth comparing.
+
+    An npm squat on a scoped package targets the part after the slash, since
+    the scope is usually owned by whoever it names.
+    """
+    lowered = name.lower()
+    if ecosystem == "npm" and lowered.startswith("@") and "/" in lowered:
+        return lowered.rsplit("/", 1)[1]
+    return lowered
+
+
+def nearest_popular(name: str, ecosystem: str = "pypi") -> tuple[str, int] | None:
     """Closest popular package name within the typo budget, if any."""
+    popular = POPULAR.get(ecosystem)
+    if not popular:
+        return None
+
     lowered = name.lower()
-    if lowered in popular:
+    target = _comparable(name, ecosystem)
+
+    if lowered in popular or target in popular:
+        return None
+    # Below this length the name space is too dense for edit distance to mean
+    # anything: 'core' sits one edit from 'cors'.
+    if len(target) < MIN_COMPARABLE_LENGTH:
         return None
 
-    budget = _typo_budget(lowered)
+    budget = _typo_budget(target)
     best: tuple[str, int] | None = None
     for candidate in popular:
-        if abs(len(candidate) - len(lowered)) > budget:
+        if abs(len(candidate) - len(target)) > budget:
             continue
-        distance = edit_distance(lowered, candidate, cutoff=budget)
+        distance = edit_distance(target, candidate, cutoff=budget)
         if 0 < distance <= budget and (best is None or distance < best[1]):
             best = (candidate, distance)
             if distance == 1:
@@ -114,7 +155,7 @@ def assess(facts: PackageFacts, strict: bool = False) -> Finding:
     is_young = facts.age_days is not None and facts.age_days < NEW_DAYS
 
     if is_young:
-        neighbour = nearest_popular(facts.name)
+        neighbour = nearest_popular(facts.name, facts.ecosystem)
         if neighbour is not None:
             popular_name, distance = neighbour
             reasons.append(

{ghostpkg-0.1.0 → ghostpkg-0.3.0}/ghostpkg/cli.py

@@ -6,11 +6,12 @@ import argparse
 import concurrent.futures
 import json
 import os
-import re
 import sys
 from pathlib import Path
 
+from . import __version__
 from .assess import Finding, Verdict, assess
+from .manifests import UnsupportedManifest, load_manifest
 from .registries import RegistryError, fetch
 
 EXIT_OK = 0
@@ -96,42 +97,14 @@ def evaluate(names: list[str], ecosystem: str, strict: bool) -> list[Finding]:
         return list(pool.map(one, names))
 
 
-REQUIREMENT_LINE = re.compile(r"^\s*([A-Za-z0-9][A-Za-z0-9._-]*)")
-
-
-def parse_requirements(text: str) -> list[str]:
-    names = []
-    for raw in text.splitlines():
-        line = raw.strip()
-        if not line or line.startswith(("#", "-", "git+", "http")):
-            continue
-        match = REQUIREMENT_LINE.match(line)
-        if match:
-            names.append(match.group(1))
-    return names
-
-
-def parse_package_json(text: str) -> list[str]:
-    data = json.loads(text)
-    names: list[str] = []
-    for key in ("dependencies", "devDependencies", "optionalDependencies"):
-        names.extend((data.get(key) or {}).keys())
-    return names
-
-
-def load_manifest(path: Path) -> tuple[list[str], str]:
-    text = path.read_text(encoding="utf-8")
-    if path.name == "package.json":
-        return parse_package_json(text), "npm"
-    return parse_requirements(text), "pypi"
-
-
 def build_parser() -> argparse.ArgumentParser:
     parser = argparse.ArgumentParser(
         prog="ghostpkg",
         description="Catch package names that do not exist before you install them.",
     )
-    parser.add_argument("--version", action="version", version="ghostpkg 0.1.0")
+    parser.add_argument(
+        "--version", action="version", version=f"ghostpkg {__version__}"
+    )
     sub = parser.add_subparsers(dest="command", required=True)
 
     check = sub.add_parser("check", help="check one or more package names")
@@ -139,7 +112,9 @@ def build_parser() -> argparse.ArgumentParser:
     check.add_argument("-e", "--ecosystem", choices=("pypi", "npm"), default="pypi")
 
     scan = sub.add_parser("scan", help="check every dependency in a manifest")
-    scan.add_argument("path", type=Path, help="requirements.txt or package.json")
+    scan.add_argument(
+        "path", type=Path, help="requirements*.txt, pyproject.toml or package.json"
+    )
 
     for command in (check, scan):
         command.add_argument(
@@ -162,7 +137,10 @@ def main(argv: list[str] | None = None) -> int:
             return EXIT_ERROR
         try:
             names, ecosystem = load_manifest(args.path)
-        except (json.JSONDecodeError, UnicodeDecodeError) as exc:
+        except UnsupportedManifest as exc:
+            print(f"ghostpkg: {exc}", file=sys.stderr)
+            return EXIT_ERROR
+        except (json.JSONDecodeError, UnicodeDecodeError, ValueError) as exc:
             print(f"ghostpkg: could not parse {args.path}: {exc}", file=sys.stderr)
             return EXIT_ERROR
         if not names: