ghostpkg 0.22.0__tar.gz → 0.23.0__tar.gz

{ghostpkg-0.22.0 → ghostpkg-0.23.0}/CHANGELOG.md

@@ -6,6 +6,44 @@ this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.23.0] - 2026-09-04
+
+Mutation testing: 186 single-line changes to the source, each run against the
+whole suite. **53 left it green.** The code was right in every case -- what was
+missing was anything that would notice if it stopped being. Writing the tests to
+kill the survivors turned up a real bug the suite had never had a chance to see.
+
+### Fixed
+- **A version in an install command made prose extraction skip the whole
+  command.** `npm install react@18` and `pip install flask==3.0` both yielded
+  nothing: the `@` was read as a direct reference and the `==` stopped the token
+  matching the name pattern. A pinned install command is the commonest form
+  there is, so the feature built to read install commands was silently skipping
+  most of them.
+
+  The version is now split off and carried through, so a hallucinated *version*
+  in a README is checked as well as a hallucinated name. Re-measured on 22 real
+  READMEs, because widening this parser has cost false positives before: 19
+  names, **none that fail to exist**.
+
+### Added
+Tests for the survivors that carry a real consequence:
+- **Each finding keeps its own file and line.** Reversing the thread pool's
+  result list left all 656 tests passing -- the printed names still looked
+  right, because a finding carries its own name, but the origin stamped on it
+  came from a different package. A CI annotation would have pointed at the
+  wrong line and marked the wrong one clean.
+- **`--deep` and `--strict` reach the scan.** Both could be forced to `False`
+  in the call without a single failure: the flag would silently do nothing.
+- **A name from prose keeps its own ecosystem**, so `npm install left-pad` in a
+  README is not looked up on PyPI.
+- Chained commands (`cd app && npm install x`), unknown installers
+  (`brew install jq`), age measured from the first upload rather than the
+  latest, `cache.put` refusing a negative, `-r base.txt  # comment` still
+  following the include, and the error counts in both output formats.
+
+693 tests. Field gate: 4,461 packages, zero blocks.
+
 ## [0.22.0] - 2026-09-04
 
 Four agents scanned **21 repositories and 78,000 package names**. They found
@@ -902,7 +940,8 @@ First release.
 - No corpus of hallucinated package names is shipped, following the decision of
   the USENIX'25 authors not to publish theirs.
 
-[Unreleased]: https://github.com/M1rwana12/ghostpkg/compare/v0.22.0...HEAD
+[Unreleased]: https://github.com/M1rwana12/ghostpkg/compare/v0.23.0...HEAD
+[0.23.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.23.0
 [0.22.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.22.0
 [0.21.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.21.0
 [0.20.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.20.0

{ghostpkg-0.22.0 → ghostpkg-0.23.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ghostpkg
-Version: 0.22.0
+Version: 0.23.0
 Summary: Catch package names that do not exist before you install them
 Project-URL: Homepage, https://github.com/m1rwana12/ghostpkg
 Project-URL: Issues, https://github.com/m1rwana12/ghostpkg/issues
@@ -441,7 +441,7 @@ length, and only applies to packages young enough to plausibly be a squat.
 ### In CI
 
 ```yaml
-- uses: M1rwana12/ghostpkg@v0.22.0
+- uses: M1rwana12/ghostpkg@v0.23.0
 ```
 
 That is the whole step. It searches the checkout, skips `node_modules` and
@@ -472,7 +472,7 @@ Or without the action, if you prefer:
 ```yaml
 repos:
   - repo: https://github.com/M1rwana12/ghostpkg
-    rev: v0.22.0
+    rev: v0.23.0
     hooks:
       - id: ghostpkg
 ```

{ghostpkg-0.22.0 → ghostpkg-0.23.0}/README.en.md

@@ -393,7 +393,7 @@ length, and only applies to packages young enough to plausibly be a squat.
 ### In CI
 
 ```yaml
-- uses: M1rwana12/ghostpkg@v0.22.0
+- uses: M1rwana12/ghostpkg@v0.23.0
 ```
 
 That is the whole step. It searches the checkout, skips `node_modules` and
@@ -424,7 +424,7 @@ Or without the action, if you prefer:
 ```yaml
 repos:
   - repo: https://github.com/M1rwana12/ghostpkg
-    rev: v0.22.0
+    rev: v0.23.0
     hooks:
       - id: ghostpkg
 ```

{ghostpkg-0.22.0 → ghostpkg-0.23.0}/README.md

@@ -190,7 +190,7 @@ ghostpkg scan ../інший-проєкт
 ### У CI
 
 ```yaml
-- uses: M1rwana12/ghostpkg@v0.22.0
+- uses: M1rwana12/ghostpkg@v0.23.0
 ```
 
 Це весь крок. Він шукає маніфести в чекауті, обходить `node_modules` і подібні
@@ -214,7 +214,7 @@ ghostpkg scan ../інший-проєкт
 ```yaml
 repos:
   - repo: https://github.com/M1rwana12/ghostpkg
-    rev: v0.22.0
+    rev: v0.23.0
     hooks:
       - id: ghostpkg
 ```

{ghostpkg-0.22.0 → ghostpkg-0.23.0}/ghostpkg/__init__.py

@@ -1,6 +1,6 @@
 """ghostpkg -- catch package names that do not exist before you install them."""
 
-__version__ = "0.22.0"
+__version__ = "0.23.0"
 
 from .assess import Finding, Verdict, assess
 from .registries import PackageFacts, RegistryError, fetch

{ghostpkg-0.22.0 → ghostpkg-0.23.0}/ghostpkg/prose.py

@@ -73,11 +73,19 @@ def looks_like_prose(filename: str) -> bool:
 SENTENCE_END = ".,;:!?)"
 
 
-def _clean(token: str) -> "tuple[str | None, bool]":
-    """(package name, whether the command ended here).
+#: Where a version starts in an install argument. npm writes `react@18`, pip
+#: writes `flask==3.0`, and both were dropped entirely: the `@` was read as a
+#: direct reference and the `==` made the token stop matching the name pattern.
+#: A pinned install command is the commonest form there is, so prose extraction
+#: was silently skipping most of what it existed to read.
+PIN = re.compile(r"(===|==|>=|<=|~=|!=|[<>@])")
 
-    Returning the second value matters more than it looks. Measured against ten
-    real READMEs, letting a command run past the end of its sentence produced a
+
+def _clean(token: str) -> "tuple[str | None, str | None, bool]":
+    """(package name, version, whether the command ended here).
+
+    The third value matters more than it looks. Measured against ten real
+    READMEs, letting a command run past the end of its sentence produced a
     **25% false-positive rate** -- `The`, `command`, `line`, `client`, `is` all
     came back as package names from one httpx line.
     """
@@ -87,22 +95,41 @@ def _clean(token: str) -> "tuple[str | None, bool]":
         token = token[:-1]
         stop = True
     if not token or token.startswith("-"):
-        return None, stop
-    # A path, a URL, an archive or a direct reference names its own source.
-    if "://" in token or token.startswith((".", "/", "~")) or "@" in token[1:]:
-        return None, stop
+        return None, None, stop
+    if "://" in token or token.startswith((".", "/", "~")):
+        return None, None, stop
     if token.endswith((".whl", ".tar.gz", ".tgz", ".zip", ".txt", ".toml", ".json")):
-        return None, stop
+        return None, None, stop
+
+    # Split the version off. A scoped npm name starts with `@`, so the search
+    # begins at the second character.
+    specifier = None
+    match = PIN.search(token, 1)
+    if match:
+        rest = token[match.start() :]
+        token, specifier = token[: match.start()], token[match.end() :].strip() or None
+        # `pkg@https://...`, `pkg@file:../x`, `pkg@owner/repo` state their own
+        # source, and the registry has nothing to say about them.
+        if rest.startswith("@") and (
+            specifier is None
+            or "://" in specifier
+            or ":" in specifier
+            or "/" in specifier
+        ):
+            return None, None, stop
+
     # `fastapi[standard]` names fastapi; the extras are not part of it.
     token = re.sub(r"\[[^\]]*\]$", "", token)
     if not token:
-        return None, stop
+        return None, None, stop
     match = NAME.match(token)
     if not match or match.group(1) != token:
-        return None, stop
+        return None, None, stop
     name = match.group(1)
     # `npm install` with no argument, or a lone dot, installs the local project.
-    return (name if name not in (".", "..") else None), stop
+    if name in (".", ".."):
+        return None, None, stop
+    return name, specifier, stop
 
 
 def _split_commands(line: str) -> list[str]:
@@ -191,7 +218,7 @@ def extract(text: str, source: str | None = None) -> list[Requirement]:
                     # An ordinary flag like --upgrade. Not a name, but not the
                     # end of the command either.
                     continue
-                name, ends_here = _clean(token)
+                name, specifier, ends_here = _clean(token)
                 if name is None:
                     # Prose has begun. Anything further on this line is English.
                     break
@@ -200,6 +227,7 @@ def extract(text: str, source: str | None = None) -> list[Requirement]:
                     found.append(
                         Requirement(
                             name=name,
+                            specifier=specifier,
                             line=number,
                             source=source,
                             ecosystem=ecosystem,

{ghostpkg-0.22.0 → ghostpkg-0.23.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "ghostpkg"
-version = "0.22.0"
+version = "0.23.0"
 description = "Catch package names that do not exist before you install them"
 readme = "README.en.md"  # English page for PyPI; the repo main page is Ukrainian
 requires-python = ">=3.9"

ghostpkg-0.23.0/tests/test_mutation_gaps.py

@@ -0,0 +1,266 @@
+"""Behaviour the suite was not defending.
+
+Found by mutation testing: 186 single-line changes to the source, each run
+against the whole suite. 53 left it green. These tests kill the survivors that
+carry a real consequence — a wrong verdict, a dropped package, a flag that
+silently does nothing.
+
+The code was correct in every case. What was missing was anything that would
+notice if it stopped being.
+"""
+
+from __future__ import annotations
+
+import json
+
+import pytest
+
+from ghostpkg.assess import Verdict
+from ghostpkg.cli import main
+from ghostpkg.manifests import parse_requirements
+from ghostpkg.prose import extract
+from ghostpkg.registries import PackageFacts
+from ghostpkg.scanner import evaluate
+
+
+def facts(name, ecosystem="pypi", **kwargs):
+    base = dict(
+        name=name, ecosystem=ecosystem, exists=True,
+        age_days=4000, release_count=30, has_repo_url=True,
+    )
+    base.update(kwargs)
+    return PackageFacts(**base)
+
+
+class TestEachFindingKeepsItsOwnOrigin:
+    """The thread pool returns results in submission order and they are zipped
+    back onto the names. Reversing that list left all 656 tests passing: the
+    printed names still looked right, because a finding carries its own name —
+    but the file and line stamped onto it came from a different package. A CI
+    annotation would point at the wrong line and mark the wrong one clean."""
+
+    def test_the_finding_named_x_carries_the_line_x_was_written_on(self, monkeypatch):
+        from ghostpkg.manifests import Requirement
+
+        monkeypatch.setattr(
+            "ghostpkg.scanner.fetch",
+            lambda name, ecosystem: facts(name, ecosystem, exists=name != "ghost"),
+        )
+        findings = evaluate(
+            [
+                Requirement(name="flask", source="requirements.txt", line=1),
+                Requirement(name="ghost", source="requirements.txt", line=2),
+                Requirement(name="requests", source="other.txt", line=7),
+            ],
+            "pypi",
+            cache=None,
+        )
+        placed = {f.name: (f.source, f.line) for f in findings}
+        assert placed["flask"] == ("requirements.txt", 1)
+        assert placed["ghost"] == ("requirements.txt", 2)
+        assert placed["requests"] == ("other.txt", 7)
+
+    def test_the_verdict_belongs_to_the_name_it_is_printed_against(self, monkeypatch):
+        monkeypatch.setattr(
+            "ghostpkg.scanner.fetch",
+            lambda name, ecosystem: facts(name, ecosystem, exists="ghost" not in name),
+        )
+        findings = evaluate(["a-real-one", "ghost-one", "b-real-one", "ghost-two"], "pypi", cache=None)
+        for finding in findings:
+            expected = Verdict.BLOCK if "ghost" in finding.name else Verdict.OK
+            assert finding.verdict is expected, finding.name
+
+    def test_order_is_preserved(self, monkeypatch):
+        monkeypatch.setattr("ghostpkg.scanner.fetch", lambda name, ecosystem: facts(name, ecosystem))
+        wanted = ["one", "two", "three", "four", "five"]
+        assert [f.name for f in evaluate(wanted, "pypi", cache=None)] == wanted
+
+
+class TestTheFlagsReachTheScan:
+    """Every flag was tested at the function it configures and none end to end
+    from `main()`. Setting `args.deep` or `args.strict` to False in the call
+    left the suite green: the flag would silently do nothing."""
+
+    def test_deep_reaches_the_scanner(self, tmp_path, monkeypatch):
+        seen = {}
+
+        def fake_evaluate(requirements, ecosystem, strict=False, cache=None, deep=False, workers=8):
+            seen["deep"] = deep
+            seen["strict"] = strict
+            return []
+
+        monkeypatch.setattr("ghostpkg.cli.evaluate", fake_evaluate)
+        path = tmp_path / "requirements.txt"
+        path.write_text("flask\n", encoding="utf-8")
+        main(["scan", str(path), "--deep", "--no-cache"])
+        assert seen["deep"] is True
+
+    def test_strict_reaches_the_scanner(self, tmp_path, monkeypatch):
+        seen = {}
+
+        def fake_evaluate(requirements, ecosystem, strict=False, cache=None, deep=False, workers=8):
+            seen["strict"] = strict
+            return []
+
+        monkeypatch.setattr("ghostpkg.cli.evaluate", fake_evaluate)
+        path = tmp_path / "requirements.txt"
+        path.write_text("flask\n", encoding="utf-8")
+        main(["scan", str(path), "--strict", "--no-cache"])
+        assert seen["strict"] is True
+
+    def test_strict_changes_the_exit_code(self, tmp_path, monkeypatch):
+        """The consequence, not just the wiring: a warning becomes a failure."""
+        monkeypatch.setattr(
+            "ghostpkg.scanner.fetch",
+            lambda name, ecosystem: facts(name, ecosystem, age_days=10, release_count=1, has_repo_url=False),
+        )
+        path = tmp_path / "requirements.txt"
+        path.write_text("something-new\n", encoding="utf-8")
+        assert main(["scan", str(path), "--no-cache"]) == 0
+        assert main(["scan", str(path), "--strict", "--no-cache"]) == 1
+
+    def test_an_error_exit_beats_a_clean_one(self, tmp_path, monkeypatch):
+        """Swapping the BLOCK/ERROR precedence survived. An unchecked name must
+        never come out as a pass."""
+        from ghostpkg.registries import RegistryError
+
+        monkeypatch.setattr(
+            "ghostpkg.scanner.fetch",
+            lambda name, ecosystem: (_ for _ in ()).throw(RegistryError("offline")),
+        )
+        path = tmp_path / "requirements.txt"
+        path.write_text("flask\n", encoding="utf-8")
+        assert main(["scan", str(path), "--no-cache"]) == 2
+
+
+class TestProseKeepsItsOwnEcosystem:
+    """A README can hold `pip install x` and `npm install y` in adjacent lines,
+    so each name carries the registry it belongs to. Dropping the per-item
+    override made every npm name from prose be looked up on PyPI -- a false
+    block on all of them, and the feature silently dead."""
+
+    def test_an_npm_name_from_prose_is_looked_up_on_npm(self, tmp_path, monkeypatch):
+        looked_up = []
+
+        def fake_fetch(name, ecosystem):
+            looked_up.append((name, ecosystem))
+            return facts(name, ecosystem)
+
+        monkeypatch.setattr("ghostpkg.scanner.fetch", fake_fetch)
+        path = tmp_path / "AGENTS.md"
+        path.write_text("Run `npm install left-pad` then `pip install flask`.\n", encoding="utf-8")
+        main(["scan", str(path), "--no-cache"])
+        assert ("left-pad", "npm") in looked_up
+        assert ("flask", "pypi") in looked_up
+
+
+class TestProseExtraction:
+    """Three survivors in one module, each verified to produce a wrong answer
+    on an ordinary README line."""
+
+    def test_a_chained_command_is_still_read(self):
+        """`_split_commands` returning the whole line dropped this entirely and
+        the run reported all clean."""
+        assert [r.name for r in extract("cd app && npm install left-pad")] == ["left-pad"]
+
+    @pytest.mark.parametrize(
+        "line",
+        ["brew install jq", "apt-get install curl nginx", "dnf install make", "cargo install ripgrep"],
+    )
+    def test_an_unknown_installer_yields_nothing(self, line):
+        """Falling through to the pip branch turned `brew install jq` into a
+        PyPI lookup -- a false block on an ordinary README."""
+        assert extract(line) == []
+
+    def test_a_pinned_npm_name_keeps_only_the_name(self):
+        """`npm install react@18` is the commonest install line there is.
+        Without the guard it was checked as the literal name `react@18`."""
+        assert [r.name for r in extract("npm install react@18")] == ["react"]
+
+
+class TestAgeComesFromTheFirstUpload:
+    """Measuring from the newest upload made every actively maintained package
+    read as days old: mass false warnings, and false blocks under `--strict`."""
+
+    def test_a_long_lived_package_is_not_young(self):
+        from ghostpkg.registries import parse_npm
+
+        payload = {
+            "name": "thing",
+            "time": {"created": "2015-01-01T00:00:00.000Z", "modified": "2026-09-01T00:00:00.000Z"},
+            "versions": {"1.0.0": {}, "2.0.0": {}},
+            "dist-tags": {"latest": "2.0.0"},
+        }
+        assert parse_npm("thing", payload).age_days > 4000
+
+
+class TestTheCacheNeverStoresAMiss:
+    """`ttl_for` is tested, the guard inside `put` was not -- so the project's
+    stated rule was only enforced second-hand."""
+
+    def test_put_refuses_a_negative(self, tmp_path):
+        from ghostpkg.cache import Cache
+
+        cache = Cache(directory=tmp_path)
+        cache.put(PackageFacts(name="ghost", ecosystem="pypi", exists=False))
+        assert cache.get("pypi", "ghost") is None
+
+    def test_put_stores_a_positive(self, tmp_path):
+        from ghostpkg.cache import Cache
+
+        cache = Cache(directory=tmp_path)
+        cache.put(facts("flask"))
+        assert cache.get("pypi", "flask") is not None
+
+
+class TestAnIncludeWithATrailingComment:
+    """Dropping the inline-comment strip stopped `-r base.txt  # shared deps`
+    following the include -- every package in the included file lost in
+    silence, with the run still reporting success."""
+
+    def test_the_include_is_followed(self, tmp_path):
+        from ghostpkg.manifests import load_manifest
+
+        (tmp_path / "base.txt").write_text("flask\njinja2\n", encoding="utf-8")
+        (tmp_path / "requirements.txt").write_text(
+            "-r base.txt  # shared deps\nrequests\n", encoding="utf-8"
+        )
+        found, _ = load_manifest(tmp_path / "requirements.txt")
+        assert set(r.name for r in found) == {"flask", "jinja2", "requests"}
+
+    def test_a_comment_after_a_name_is_stripped(self):
+        assert [r.name for r in parse_requirements("flask  # web\n")] == ["flask"]
+
+
+class TestTheErrorSummaryIsReported:
+    """The error count in `--json` and the "could not be checked" line both
+    survived mutation. A run that could not reach the registry must not read as
+    a clean one."""
+
+    def test_json_counts_the_errors(self, tmp_path, monkeypatch, capsys):
+        from ghostpkg.registries import RegistryError
+
+        monkeypatch.setattr(
+            "ghostpkg.scanner.fetch",
+            lambda name, ecosystem: (_ for _ in ()).throw(RegistryError("offline")),
+        )
+        path = tmp_path / "requirements.txt"
+        path.write_text("flask\n", encoding="utf-8")
+        main(["scan", str(path), "--no-cache", "--json"])
+        payload = json.loads(capsys.readouterr().out)
+        assert payload["summary"]["errored"] == 1
+        assert payload["summary"]["blocked"] == 0
+
+    def test_the_text_summary_says_so(self, tmp_path, monkeypatch, capsys):
+        from ghostpkg.registries import RegistryError
+
+        monkeypatch.setattr(
+            "ghostpkg.scanner.fetch",
+            lambda name, ecosystem: (_ for _ in ()).throw(RegistryError("offline")),
+        )
+        path = tmp_path / "requirements.txt"
+        path.write_text("flask\n", encoding="utf-8")
+        main(["scan", str(path), "--no-cache"])
+        out = capsys.readouterr().out
+        assert "could not be checked" in out
+        assert "look fine" not in out

{ghostpkg-0.22.0 → ghostpkg-0.23.0}/tests/test_prose.py

@@ -198,3 +198,59 @@ class TestACommandInsideASentence:
 
     def test_a_span_with_a_backtick_pair_but_no_installer(self):
         assert self.names("The `requirements.txt` file lists them.") == []
+
+
+class TestAVersionInAnInstallCommand:
+    """`npm install react@18` and `pip install flask==3.0` both extracted
+    nothing at all. The `@` was read as a direct reference and the `==` made the
+    token stop matching the name pattern, so a pinned install command -- the
+    commonest form there is -- was silently skipped by the feature built to
+    read install commands.
+
+    Re-measured after the change, because widening this parser has cost false
+    positives before: 19 names across 22 real READMEs, none that fail to exist.
+    """
+
+    @pytest.mark.parametrize(
+        "line, name, specifier",
+        [
+            ("npm install react@18", "react", "18"),
+            ("npm i lodash@^4.17.21", "lodash", "^4.17.21"),
+            ("pnpm add vue@3", "vue", "3"),
+            ("yarn add @babel/core@7", "@babel/core", "7"),
+            ("bun add hono@4.0.0", "hono", "4.0.0"),
+            ("pip install flask==3.0", "flask", "3.0"),
+            ("pip install httpx>=0.27", "httpx", "0.27"),
+            ("pip install django~=5.0", "django", "5.0"),
+            ("uv add fastapi==0.115.0", "fastapi", "0.115.0"),
+            ("npm install react", "react", None),
+        ],
+    )
+    def test_the_name_and_the_version_are_both_read(self, line, name, specifier):
+        found = extract(line)
+        assert [(r.name, r.specifier) for r in found] == [(name, specifier)]
+
+    @pytest.mark.parametrize(
+        "line",
+        [
+            "npm install pkg@https://example.com/x.tgz",
+            "npm install pkg@file:../local",
+            "npm install pkg@acme/repo",
+            "npm install pkg@github:acme/repo",
+        ],
+    )
+    def test_a_stated_source_after_the_at_is_still_skipped(self, line):
+        """The `@` guard existed to catch these. Splitting the version off must
+        not let a direct reference through."""
+        assert extract(line) == []
+
+    def test_a_pinned_name_in_prose_keeps_its_line(self):
+        text = "intro\n\nRun `pip install ghost-thing==1.0` to begin.\n"
+        found = extract(text)
+        assert (found[0].name, found[0].specifier, found[0].line) == ("ghost-thing", "1.0", 3)
+
+    def test_the_sentence_still_ends_the_command(self):
+        """The 25% false-positive rate this parser once had came from running
+        past the full stop. Splitting versions must not reopen that."""
+        text = "Run `pip install httpx==0.27` and the command line client is optional."
+        assert [r.name for r in extract(text)] == ["httpx"]