ghostpkg 0.22.0__tar.gz → 0.24.0__tar.gz

ghostpkg-0.24.0/.github/workflows/field.yml

@@ -0,0 +1,71 @@
+# Scanning real repositories, on a schedule rather than on every push.
+#
+# This gate has found a defect the unit tests missed on three consecutive runs,
+# including two that reported a package as non-existent while `pip` installed it
+# without complaint. It is the most productive check the project has, and it is
+# the one nobody would remember to run by hand.
+#
+# It is not on every push because a full pass is roughly 40,000 registry
+# lookups. That is rude to PyPI and npm, and the answers barely move day to day.
+# The push job runs the three smallest repositories, which is about 700 names
+# and mostly served from cache.
+#
+# Runs on Linux specifically: `vercel/next.js` cannot be cloned on Windows,
+# where a path in its test fixtures exceeds the limit, so the largest monorepo
+# in the set is only reachable here.
+
+name: field
+
+on:
+  schedule:
+    # 04:20 UTC, off the hour to avoid the stampede every cron service creates.
+    - cron: "20 4 * * *"
+  workflow_dispatch:
+    inputs:
+      quick:
+        description: Only the three smallest repositories
+        type: boolean
+        default: false
+  push:
+    branches: [main]
+    paths:
+      # Only when something that could change a verdict has changed.
+      - "ghostpkg/**"
+      - "scripts/fieldtest.py"
+      - ".github/workflows/field.yml"
+
+permissions:
+  contents: read
+
+jobs:
+  scan:
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - uses: actions/checkout@v7
+
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.12"
+
+      - run: pip install -e .
+
+      - name: Scan real repositories
+        run: |
+          # A push gets the quick set; a schedule or a manual run gets all of
+          # them, unless the manual run asked for quick.
+          if [ "${{ github.event_name }}" = "push" ] || [ "${{ inputs.quick }}" = "true" ]; then
+            python scripts/fieldtest.py --quick
+          else
+            python scripts/fieldtest.py
+          fi
+
+      - name: What a failure here means
+        if: failure()
+        run: |
+          echo "::error::A block was reported on a repository whose dependencies"
+          echo "::error::real people install every day. Treat it as a false"
+          echo "::error::positive until proven otherwise: find where the name"
+          echo "::error::came from, fix the parser, and re-run. If the block is"
+          echo "::error::genuinely correct, add it to EXPECTED in"
+          echo "::error::scripts/fieldtest.py with a reason a reader can check."

{ghostpkg-0.22.0 → ghostpkg-0.24.0}/CHANGELOG.md

@@ -6,6 +6,87 @@ this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.24.0] - 2026-09-04
+
+The gate now runs in CI, and it scanned **88,904 packages across sixteen
+repositories**. Six blocks, and **all six are real** -- unsatisfiable pins in
+Ray and a package next.js references that was never published. Zero false
+positives at that scale is the number this project exists to hold.
+
+### Added
+- **A scope the project already leans on is no longer treated as a stranger.**
+  Almost all remaining noise was "published under a year ago" on the dozen
+  platform binaries a compiled tool ships at once:
+  `@oxfmt/binding-darwin-arm64`, `@oxc-parser/binding-linux-riscv64-musl`.
+
+  The justification is structural rather than statistical: **an npm scope is
+  owned**. To publish `@oxfmt/anything` you must control `@oxfmt`, so a young
+  package in a scope the scan already depends on three times over is the same
+  publisher shipping another build.
+
+  Measured before it was built -- two members silences 41% of warnings, three
+  silences 40%, six silences 39%, so the threshold sits where it is defensible
+  rather than where it is largest. In practice: Sentry **114 warnings to 22**,
+  Vue **15 to 5**, and no change whatever to the block count. A squatted scope
+  (`@types-node` is not `@types/node`) appears once in a scan, not three times,
+  which is what the threshold separates. Nothing that blocks is affected, and
+  neither is the lookalike check.
+
+- **`.github/workflows/field.yml`** runs the gate nightly and on any push that
+  touches the scanner. On Linux, because `vercel/next.js` cannot be cloned on
+  Windows -- a path in its test fixtures exceeds the limit, so the largest
+  monorepo in the set was unreachable until now.
+
+- The repository list is **sixteen**, each carrying a note saying which defect
+  it found, so a future reader can tell why the list is what it is instead of
+  trimming it for being slow.
+
+### Verified
+`EXPECTED` now records the three names that genuinely do not exist, each
+checked against the registry by hand: `jaxlib==0.4.17` (PyPI's oldest is
+0.4.18), `tensorflow-macos==2.20.0` (Apple stopped at 2.16.2), and
+`tsconfig-mod` (404 on npm).
+
+707 tests.
+
+## [0.23.0] - 2026-09-04
+
+Mutation testing: 186 single-line changes to the source, each run against the
+whole suite. **53 left it green.** The code was right in every case -- what was
+missing was anything that would notice if it stopped being. Writing the tests to
+kill the survivors turned up a real bug the suite had never had a chance to see.
+
+### Fixed
+- **A version in an install command made prose extraction skip the whole
+  command.** `npm install react@18` and `pip install flask==3.0` both yielded
+  nothing: the `@` was read as a direct reference and the `==` stopped the token
+  matching the name pattern. A pinned install command is the commonest form
+  there is, so the feature built to read install commands was silently skipping
+  most of them.
+
+  The version is now split off and carried through, so a hallucinated *version*
+  in a README is checked as well as a hallucinated name. Re-measured on 22 real
+  READMEs, because widening this parser has cost false positives before: 19
+  names, **none that fail to exist**.
+
+### Added
+Tests for the survivors that carry a real consequence:
+- **Each finding keeps its own file and line.** Reversing the thread pool's
+  result list left all 656 tests passing -- the printed names still looked
+  right, because a finding carries its own name, but the origin stamped on it
+  came from a different package. A CI annotation would have pointed at the
+  wrong line and marked the wrong one clean.
+- **`--deep` and `--strict` reach the scan.** Both could be forced to `False`
+  in the call without a single failure: the flag would silently do nothing.
+- **A name from prose keeps its own ecosystem**, so `npm install left-pad` in a
+  README is not looked up on PyPI.
+- Chained commands (`cd app && npm install x`), unknown installers
+  (`brew install jq`), age measured from the first upload rather than the
+  latest, `cache.put` refusing a negative, `-r base.txt  # comment` still
+  following the include, and the error counts in both output formats.
+
+693 tests. Field gate: 4,461 packages, zero blocks.
+
 ## [0.22.0] - 2026-09-04
 
 Four agents scanned **21 repositories and 78,000 package names**. They found
@@ -902,7 +983,9 @@ First release.
 - No corpus of hallucinated package names is shipped, following the decision of
   the USENIX'25 authors not to publish theirs.
 
-[Unreleased]: https://github.com/M1rwana12/ghostpkg/compare/v0.22.0...HEAD
+[Unreleased]: https://github.com/M1rwana12/ghostpkg/compare/v0.24.0...HEAD
+[0.24.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.24.0
+[0.23.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.23.0
 [0.22.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.22.0
 [0.21.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.21.0
 [0.20.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.20.0

{ghostpkg-0.22.0 → ghostpkg-0.24.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ghostpkg
-Version: 0.22.0
+Version: 0.24.0
 Summary: Catch package names that do not exist before you install them
 Project-URL: Homepage, https://github.com/m1rwana12/ghostpkg
 Project-URL: Issues, https://github.com/m1rwana12/ghostpkg/issues
@@ -441,7 +441,7 @@ length, and only applies to packages young enough to plausibly be a squat.
 ### In CI
 
 ```yaml
-- uses: M1rwana12/ghostpkg@v0.22.0
+- uses: M1rwana12/ghostpkg@v0.24.0
 ```
 
 That is the whole step. It searches the checkout, skips `node_modules` and
@@ -472,7 +472,7 @@ Or without the action, if you prefer:
 ```yaml
 repos:
   - repo: https://github.com/M1rwana12/ghostpkg
-    rev: v0.22.0
+    rev: v0.24.0
     hooks:
       - id: ghostpkg
 ```

{ghostpkg-0.22.0 → ghostpkg-0.24.0}/README.en.md

@@ -393,7 +393,7 @@ length, and only applies to packages young enough to plausibly be a squat.
 ### In CI
 
 ```yaml
-- uses: M1rwana12/ghostpkg@v0.22.0
+- uses: M1rwana12/ghostpkg@v0.24.0
 ```
 
 That is the whole step. It searches the checkout, skips `node_modules` and
@@ -424,7 +424,7 @@ Or without the action, if you prefer:
 ```yaml
 repos:
   - repo: https://github.com/M1rwana12/ghostpkg
-    rev: v0.22.0
+    rev: v0.24.0
     hooks:
       - id: ghostpkg
 ```

{ghostpkg-0.22.0 → ghostpkg-0.24.0}/README.md

@@ -190,7 +190,7 @@ ghostpkg scan ../інший-проєкт
 ### У CI
 
 ```yaml
-- uses: M1rwana12/ghostpkg@v0.22.0
+- uses: M1rwana12/ghostpkg@v0.24.0
 ```
 
 Це весь крок. Він шукає маніфести в чекауті, обходить `node_modules` і подібні
@@ -214,7 +214,7 @@ ghostpkg scan ../інший-проєкт
 ```yaml
 repos:
   - repo: https://github.com/M1rwana12/ghostpkg
-    rev: v0.22.0
+    rev: v0.24.0
     hooks:
       - id: ghostpkg
 ```

{ghostpkg-0.22.0 → ghostpkg-0.24.0}/ghostpkg/__init__.py

@@ -1,6 +1,6 @@
 """ghostpkg -- catch package names that do not exist before you install them."""
 
-__version__ = "0.22.0"
+__version__ = "0.24.0"
 
 from .assess import Finding, Verdict, assess
 from .registries import PackageFacts, RegistryError, fetch

{ghostpkg-0.22.0 → ghostpkg-0.24.0}/ghostpkg/assess.py

@@ -216,11 +216,48 @@ def nearest_popular(name: str, ecosystem: str = "pypi") -> tuple[str, int] | Non
     return best
 
 
+#: How many packages a scope needs in one scan before its young members stop
+#: being warned about. Measured across four repositories: two silences 41% of
+#: all warnings, three silences 40%, six silences 39%. The effect saturates
+#: immediately, so the threshold sits where it is defensible rather than where
+#: it is largest.
+TRUSTED_SCOPE_MEMBERS = 3
+
+
+def trusted_scopes(names: "list[str]") -> "set[str]":
+    """npm scopes this scan depends on several times over.
+
+    The justification is structural rather than statistical: **an npm scope is
+    owned**. To publish `@oxfmt/binding-darwin-arm64` you must control
+    `@oxfmt`. So a young package in a scope the project already uses three
+    times is the same publisher shipping another build -- usually one of the
+    dozen platform binaries a compiled tool releases at once -- and not a
+    stranger who guessed a name.
+
+    A scope is itself squattable: `@types-node` is not `@types/node`. That is
+    the risk this has to survive, and the threshold is what does it -- a
+    squatted scope appears once in a scan, not three times. An attacker who
+    genuinely owned a scope a project leaned on that heavily would have far
+    better attacks available than guessing one more name inside it.
+
+    Measured: 106 of 268 warnings across four popular repositories, every one
+    of them a platform binary, and none of them something a person would want
+    to be told about.
+    """
+    counts: dict[str, int] = {}
+    for name in names:
+        if name.startswith("@") and "/" in name:
+            scope = name.split("/", 1)[0]
+            counts[scope] = counts.get(scope, 0) + 1
+    return {scope for scope, n in counts.items() if n >= TRUSTED_SCOPE_MEMBERS}
+
+
 def assess(
     facts: PackageFacts,
     strict: bool = False,
     signals: "list | None" = None,
     specifier: str | None = None,
+    known_scope: bool = False,
 ) -> Finding:
     """Turn registry facts, and optionally --deep install-script signals, into
     a verdict.
@@ -304,6 +341,13 @@ def assess(
 
     reasons: list[Reason] = []
 
+    # `known_scope` says the scan already depends on this scope several times,
+    # so "new and thin" describes a known publisher rather than a stranger.
+    # It quiets the soft signals only. Nothing that blocks is affected, and
+    # neither is the lookalike check -- a typo inside a trusted scope is still
+    # worth saying.
+    soft = not known_scope
+
     # A withdrawn version exists, so it is not a block -- and pip installs one
     # anyway when it is pinned explicitly, so blocking would be stricter than
     # the package manager itself. But the maintainer said not to use it, and
@@ -320,7 +364,7 @@ def assess(
                 )
             )
 
-    if facts.age_days is not None:
+    if soft and facts.age_days is not None:
         if facts.age_days < YOUNG_DAYS:
             reasons.append(
                 Reason(GP_RECENT, f"first published {facts.age_days} days ago")
@@ -361,10 +405,10 @@ def assess(
                 )
             )
 
-    if is_young and facts.release_count <= 1:
+    if soft and is_young and facts.release_count <= 1:
         reasons.append(Reason(GP_ONE_RELEASE, "only one release"))
 
-    if is_young and not facts.has_repo_url:
+    if soft and is_young and not facts.has_repo_url:
         reasons.append(Reason(GP_NO_REPO, "no repository or homepage link"))
 
     install_reasons = [

{ghostpkg-0.22.0 → ghostpkg-0.24.0}/ghostpkg/prose.py

@@ -73,11 +73,19 @@ def looks_like_prose(filename: str) -> bool:
 SENTENCE_END = ".,;:!?)"
 
 
-def _clean(token: str) -> "tuple[str | None, bool]":
-    """(package name, whether the command ended here).
+#: Where a version starts in an install argument. npm writes `react@18`, pip
+#: writes `flask==3.0`, and both were dropped entirely: the `@` was read as a
+#: direct reference and the `==` made the token stop matching the name pattern.
+#: A pinned install command is the commonest form there is, so prose extraction
+#: was silently skipping most of what it existed to read.
+PIN = re.compile(r"(===|==|>=|<=|~=|!=|[<>@])")
 
-    Returning the second value matters more than it looks. Measured against ten
-    real READMEs, letting a command run past the end of its sentence produced a
+
+def _clean(token: str) -> "tuple[str | None, str | None, bool]":
+    """(package name, version, whether the command ended here).
+
+    The third value matters more than it looks. Measured against ten real
+    READMEs, letting a command run past the end of its sentence produced a
     **25% false-positive rate** -- `The`, `command`, `line`, `client`, `is` all
     came back as package names from one httpx line.
     """
@@ -87,22 +95,41 @@ def _clean(token: str) -> "tuple[str | None, bool]":
         token = token[:-1]
         stop = True
     if not token or token.startswith("-"):
-        return None, stop
-    # A path, a URL, an archive or a direct reference names its own source.
-    if "://" in token or token.startswith((".", "/", "~")) or "@" in token[1:]:
-        return None, stop
+        return None, None, stop
+    if "://" in token or token.startswith((".", "/", "~")):
+        return None, None, stop
     if token.endswith((".whl", ".tar.gz", ".tgz", ".zip", ".txt", ".toml", ".json")):
-        return None, stop
+        return None, None, stop
+
+    # Split the version off. A scoped npm name starts with `@`, so the search
+    # begins at the second character.
+    specifier = None
+    match = PIN.search(token, 1)
+    if match:
+        rest = token[match.start() :]
+        token, specifier = token[: match.start()], token[match.end() :].strip() or None
+        # `pkg@https://...`, `pkg@file:../x`, `pkg@owner/repo` state their own
+        # source, and the registry has nothing to say about them.
+        if rest.startswith("@") and (
+            specifier is None
+            or "://" in specifier
+            or ":" in specifier
+            or "/" in specifier
+        ):
+            return None, None, stop
+
     # `fastapi[standard]` names fastapi; the extras are not part of it.
     token = re.sub(r"\[[^\]]*\]$", "", token)
     if not token:
-        return None, stop
+        return None, None, stop
     match = NAME.match(token)
     if not match or match.group(1) != token:
-        return None, stop
+        return None, None, stop
     name = match.group(1)
     # `npm install` with no argument, or a lone dot, installs the local project.
-    return (name if name not in (".", "..") else None), stop
+    if name in (".", ".."):
+        return None, None, stop
+    return name, specifier, stop
 
 
 def _split_commands(line: str) -> list[str]:
@@ -191,7 +218,7 @@ def extract(text: str, source: str | None = None) -> list[Requirement]:
                     # An ordinary flag like --upgrade. Not a name, but not the
                     # end of the command either.
                     continue
-                name, ends_here = _clean(token)
+                name, specifier, ends_here = _clean(token)
                 if name is None:
                     # Prose has begun. Anything further on this line is English.
                     break
@@ -200,6 +227,7 @@ def extract(text: str, source: str | None = None) -> list[Requirement]:
                     found.append(
                         Requirement(
                             name=name,
+                            specifier=specifier,
                             line=number,
                             source=source,
                             ecosystem=ecosystem,

{ghostpkg-0.22.0 → ghostpkg-0.24.0}/ghostpkg/scanner.py

@@ -9,7 +9,7 @@ from __future__ import annotations
 import concurrent.futures
 from dataclasses import replace
 
-from .assess import NEW_DAYS, Finding, Verdict, assess, exact_pin
+from .assess import NEW_DAYS, Finding, Verdict, assess, exact_pin, trusted_scopes
 from .cache import Cache
 from .inspection import InspectionError, inspect_package
 from .manifests import Requirement
@@ -30,6 +30,7 @@ def evaluate(
     items = [
         r if isinstance(r, Requirement) else Requirement(name=r) for r in requirements
     ]
+    scopes = trusted_scopes([item.name for item in items])
 
     def one(requirement: Requirement) -> Finding:
         name = requirement.name
@@ -94,6 +95,7 @@ def evaluate(
             # A constraints entry is a bound, not an install, so its pin is not
             # a claim that such a version exists. The name is still checked.
             specifier=None if requirement.constraint else requirement.specifier,
+            known_scope=name.split("/", 1)[0] in scopes if name.startswith("@") else False,
         )
         finding.source = requirement.source
         finding.line = requirement.line

{ghostpkg-0.22.0 → ghostpkg-0.24.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "ghostpkg"
-version = "0.22.0"
+version = "0.24.0"
 description = "Catch package names that do not exist before you install them"
 readme = "README.en.md"  # English page for PyPI; the repo main page is Ukrainian
 requires-python = ">=3.9"

{ghostpkg-0.22.0 → ghostpkg-0.24.0}/scripts/fieldtest.py

@@ -29,20 +29,51 @@ import tempfile
 import time
 from pathlib import Path
 
-#: Chosen for the shapes they contain, not for popularity: a pnpm monorepo with
-#: hundreds of workspace members, a Python project with pinned calendar
-#: versions, and a repository that mixes both ecosystems in one tree.
+#: Chosen for the shapes they contain, not for popularity. Every one of these
+#: found a defect nothing else did, and the note says which -- so a future
+#: reader can tell why the list is what it is rather than trimming it.
+#:
+#: The boolean is whether `--quick` includes it.
 REPOS = [
-    ("pallets/flask", "uv.lock and example sub-projects", True),
+    ("pallets/flask", "uv.lock, example sub-projects", True),
     ("vuejs/core", "pnpm workspace, lockfile v9", True),
+    ("expressjs/express", "small and plain npm; a fast canary", True),
+
     ("getsentry/sentry", "pnpm-lock and uv.lock in one tree", False),
-    ("home-assistant/core", "calendar versions with leading zeros", False),
+    ("home-assistant/core", "constraints pinned to 1000000000.0.0; calendar versions", False),
+    ("vercel/turborepo", "pnpm v5 keys with peer suffixes -- 17 false blocks", False),
+    ("sveltejs/kit", "pnpm `name@file:` workspace fixtures -- 5 false blocks", False),
+    ("withastro/astro", "more `name@file:` fixtures, scoped -- 3 false blocks", False),
+    ("storybookjs/storybook", "yarn berry `npm:v1.1.0` ranges -- 2 false blocks", False),
+    ("apache/airflow", "127 checksum files, five named like constraints", False),
+    ("ray-project/ray", "PEP 440 local versions, materialised symlinks", False),
+    ("nrwl/nx", "pnpm monorepo, agent instruction files", False),
+    ("angular/angular", "15,000 names, the widest single scan", False),
+    ("jupyterlab/jupyterlab", "Python and npm in one tree", False),
+    ("apache/superset", "6,000 names, heavy extras", False),
     ("vercel/next.js", "491 package.json files, internal deps by exact pin", False),
 ]
 
 #: Blocks that are correct. A repository may genuinely depend on something that
-#: no longer exists; each entry needs a reason a reader can check.
-EXPECTED: dict[str, str] = {}
+#: does not exist, and every entry here was verified against the registry by
+#: hand -- the reason has to be checkable by the next reader.
+#:
+#: Across 88,904 packages in sixteen repositories these six are the only blocks,
+#: and all six are real. That is the number the gate exists to hold.
+EXPECTED = {
+    # PyPI's jaxlib release list begins at 0.4.18. Older wheels were served
+    # from Google storage and the PyPI records were removed, so
+    # `pip install -r` on that file fails today.
+    "jaxlib": "ray pins 0.4.17; PyPI's oldest jaxlib release is 0.4.18",
+
+    # Apple stopped publishing tensorflow-macos at 2.16.2. Ray copied the
+    # `tensorflow==2.20.0` pin onto the macOS package, guarded by a
+    # `sys_platform == 'darwin'` marker, so it only fails on a Mac.
+    "tensorflow-macos": "ray pins 2.20.0; PyPI's newest is 2.16.2",
+
+    # 404 on npmjs.
+    "tsconfig-mod": "referenced by next.js and not published on npm",
+}
 
 CACHE = Path(tempfile.gettempdir()) / "ghostpkg-fieldtest"
 

ghostpkg-0.24.0/tests/test_mutation_gaps.py

@@ -0,0 +1,266 @@
+"""Behaviour the suite was not defending.
+
+Found by mutation testing: 186 single-line changes to the source, each run
+against the whole suite. 53 left it green. These tests kill the survivors that
+carry a real consequence — a wrong verdict, a dropped package, a flag that
+silently does nothing.
+
+The code was correct in every case. What was missing was anything that would
+notice if it stopped being.
+"""
+
+from __future__ import annotations
+
+import json
+
+import pytest
+
+from ghostpkg.assess import Verdict
+from ghostpkg.cli import main
+from ghostpkg.manifests import parse_requirements
+from ghostpkg.prose import extract
+from ghostpkg.registries import PackageFacts
+from ghostpkg.scanner import evaluate
+
+
+def facts(name, ecosystem="pypi", **kwargs):
+    base = dict(
+        name=name, ecosystem=ecosystem, exists=True,
+        age_days=4000, release_count=30, has_repo_url=True,
+    )
+    base.update(kwargs)
+    return PackageFacts(**base)
+
+
+class TestEachFindingKeepsItsOwnOrigin:
+    """The thread pool returns results in submission order and they are zipped
+    back onto the names. Reversing that list left all 656 tests passing: the
+    printed names still looked right, because a finding carries its own name —
+    but the file and line stamped onto it came from a different package. A CI
+    annotation would point at the wrong line and mark the wrong one clean."""
+
+    def test_the_finding_named_x_carries_the_line_x_was_written_on(self, monkeypatch):
+        from ghostpkg.manifests import Requirement
+
+        monkeypatch.setattr(
+            "ghostpkg.scanner.fetch",
+            lambda name, ecosystem: facts(name, ecosystem, exists=name != "ghost"),
+        )
+        findings = evaluate(
+            [
+                Requirement(name="flask", source="requirements.txt", line=1),
+                Requirement(name="ghost", source="requirements.txt", line=2),
+                Requirement(name="requests", source="other.txt", line=7),
+            ],
+            "pypi",
+            cache=None,
+        )
+        placed = {f.name: (f.source, f.line) for f in findings}
+        assert placed["flask"] == ("requirements.txt", 1)
+        assert placed["ghost"] == ("requirements.txt", 2)
+        assert placed["requests"] == ("other.txt", 7)
+
+    def test_the_verdict_belongs_to_the_name_it_is_printed_against(self, monkeypatch):
+        monkeypatch.setattr(
+            "ghostpkg.scanner.fetch",
+            lambda name, ecosystem: facts(name, ecosystem, exists="ghost" not in name),
+        )
+        findings = evaluate(["a-real-one", "ghost-one", "b-real-one", "ghost-two"], "pypi", cache=None)
+        for finding in findings:
+            expected = Verdict.BLOCK if "ghost" in finding.name else Verdict.OK
+            assert finding.verdict is expected, finding.name
+
+    def test_order_is_preserved(self, monkeypatch):
+        monkeypatch.setattr("ghostpkg.scanner.fetch", lambda name, ecosystem: facts(name, ecosystem))
+        wanted = ["one", "two", "three", "four", "five"]
+        assert [f.name for f in evaluate(wanted, "pypi", cache=None)] == wanted
+
+
+class TestTheFlagsReachTheScan:
+    """Every flag was tested at the function it configures and none end to end
+    from `main()`. Setting `args.deep` or `args.strict` to False in the call
+    left the suite green: the flag would silently do nothing."""
+
+    def test_deep_reaches_the_scanner(self, tmp_path, monkeypatch):
+        seen = {}
+
+        def fake_evaluate(requirements, ecosystem, strict=False, cache=None, deep=False, workers=8):
+            seen["deep"] = deep
+            seen["strict"] = strict
+            return []
+
+        monkeypatch.setattr("ghostpkg.cli.evaluate", fake_evaluate)
+        path = tmp_path / "requirements.txt"
+        path.write_text("flask\n", encoding="utf-8")
+        main(["scan", str(path), "--deep", "--no-cache"])
+        assert seen["deep"] is True
+
+    def test_strict_reaches_the_scanner(self, tmp_path, monkeypatch):
+        seen = {}
+
+        def fake_evaluate(requirements, ecosystem, strict=False, cache=None, deep=False, workers=8):
+            seen["strict"] = strict
+            return []
+
+        monkeypatch.setattr("ghostpkg.cli.evaluate", fake_evaluate)
+        path = tmp_path / "requirements.txt"
+        path.write_text("flask\n", encoding="utf-8")
+        main(["scan", str(path), "--strict", "--no-cache"])
+        assert seen["strict"] is True
+
+    def test_strict_changes_the_exit_code(self, tmp_path, monkeypatch):
+        """The consequence, not just the wiring: a warning becomes a failure."""
+        monkeypatch.setattr(
+            "ghostpkg.scanner.fetch",
+            lambda name, ecosystem: facts(name, ecosystem, age_days=10, release_count=1, has_repo_url=False),
+        )
+        path = tmp_path / "requirements.txt"
+        path.write_text("something-new\n", encoding="utf-8")
+        assert main(["scan", str(path), "--no-cache"]) == 0
+        assert main(["scan", str(path), "--strict", "--no-cache"]) == 1
+
+    def test_an_error_exit_beats_a_clean_one(self, tmp_path, monkeypatch):
+        """Swapping the BLOCK/ERROR precedence survived. An unchecked name must
+        never come out as a pass."""
+        from ghostpkg.registries import RegistryError
+
+        monkeypatch.setattr(
+            "ghostpkg.scanner.fetch",
+            lambda name, ecosystem: (_ for _ in ()).throw(RegistryError("offline")),
+        )
+        path = tmp_path / "requirements.txt"
+        path.write_text("flask\n", encoding="utf-8")
+        assert main(["scan", str(path), "--no-cache"]) == 2
+
+
+class TestProseKeepsItsOwnEcosystem:
+    """A README can hold `pip install x` and `npm install y` in adjacent lines,
+    so each name carries the registry it belongs to. Dropping the per-item
+    override made every npm name from prose be looked up on PyPI -- a false
+    block on all of them, and the feature silently dead."""
+
+    def test_an_npm_name_from_prose_is_looked_up_on_npm(self, tmp_path, monkeypatch):
+        looked_up = []
+
+        def fake_fetch(name, ecosystem):
+            looked_up.append((name, ecosystem))
+            return facts(name, ecosystem)
+
+        monkeypatch.setattr("ghostpkg.scanner.fetch", fake_fetch)
+        path = tmp_path / "AGENTS.md"
+        path.write_text("Run `npm install left-pad` then `pip install flask`.\n", encoding="utf-8")
+        main(["scan", str(path), "--no-cache"])
+        assert ("left-pad", "npm") in looked_up
+        assert ("flask", "pypi") in looked_up
+
+
+class TestProseExtraction:
+    """Three survivors in one module, each verified to produce a wrong answer
+    on an ordinary README line."""
+
+    def test_a_chained_command_is_still_read(self):
+        """`_split_commands` returning the whole line dropped this entirely and
+        the run reported all clean."""
+        assert [r.name for r in extract("cd app && npm install left-pad")] == ["left-pad"]
+
+    @pytest.mark.parametrize(
+        "line",
+        ["brew install jq", "apt-get install curl nginx", "dnf install make", "cargo install ripgrep"],
+    )
+    def test_an_unknown_installer_yields_nothing(self, line):
+        """Falling through to the pip branch turned `brew install jq` into a
+        PyPI lookup -- a false block on an ordinary README."""
+        assert extract(line) == []
+
+    def test_a_pinned_npm_name_keeps_only_the_name(self):
+        """`npm install react@18` is the commonest install line there is.
+        Without the guard it was checked as the literal name `react@18`."""
+        assert [r.name for r in extract("npm install react@18")] == ["react"]
+
+
+class TestAgeComesFromTheFirstUpload:
+    """Measuring from the newest upload made every actively maintained package
+    read as days old: mass false warnings, and false blocks under `--strict`."""
+
+    def test_a_long_lived_package_is_not_young(self):
+        from ghostpkg.registries import parse_npm
+
+        payload = {
+            "name": "thing",
+            "time": {"created": "2015-01-01T00:00:00.000Z", "modified": "2026-09-01T00:00:00.000Z"},
+            "versions": {"1.0.0": {}, "2.0.0": {}},
+            "dist-tags": {"latest": "2.0.0"},
+        }
+        assert parse_npm("thing", payload).age_days > 4000
+
+
+class TestTheCacheNeverStoresAMiss:
+    """`ttl_for` is tested, the guard inside `put` was not -- so the project's
+    stated rule was only enforced second-hand."""
+
+    def test_put_refuses_a_negative(self, tmp_path):
+        from ghostpkg.cache import Cache
+
+        cache = Cache(directory=tmp_path)
+        cache.put(PackageFacts(name="ghost", ecosystem="pypi", exists=False))
+        assert cache.get("pypi", "ghost") is None
+
+    def test_put_stores_a_positive(self, tmp_path):
+        from ghostpkg.cache import Cache
+
+        cache = Cache(directory=tmp_path)
+        cache.put(facts("flask"))
+        assert cache.get("pypi", "flask") is not None
+
+
+class TestAnIncludeWithATrailingComment:
+    """Dropping the inline-comment strip stopped `-r base.txt  # shared deps`
+    following the include -- every package in the included file lost in
+    silence, with the run still reporting success."""
+
+    def test_the_include_is_followed(self, tmp_path):
+        from ghostpkg.manifests import load_manifest
+
+        (tmp_path / "base.txt").write_text("flask\njinja2\n", encoding="utf-8")
+        (tmp_path / "requirements.txt").write_text(
+            "-r base.txt  # shared deps\nrequests\n", encoding="utf-8"
+        )
+        found, _ = load_manifest(tmp_path / "requirements.txt")
+        assert set(r.name for r in found) == {"flask", "jinja2", "requests"}
+
+    def test_a_comment_after_a_name_is_stripped(self):
+        assert [r.name for r in parse_requirements("flask  # web\n")] == ["flask"]
+
+
+class TestTheErrorSummaryIsReported:
+    """The error count in `--json` and the "could not be checked" line both
+    survived mutation. A run that could not reach the registry must not read as
+    a clean one."""
+
+    def test_json_counts_the_errors(self, tmp_path, monkeypatch, capsys):
+        from ghostpkg.registries import RegistryError
+
+        monkeypatch.setattr(
+            "ghostpkg.scanner.fetch",
+            lambda name, ecosystem: (_ for _ in ()).throw(RegistryError("offline")),
+        )
+        path = tmp_path / "requirements.txt"
+        path.write_text("flask\n", encoding="utf-8")
+        main(["scan", str(path), "--no-cache", "--json"])
+        payload = json.loads(capsys.readouterr().out)
+        assert payload["summary"]["errored"] == 1
+        assert payload["summary"]["blocked"] == 0
+
+    def test_the_text_summary_says_so(self, tmp_path, monkeypatch, capsys):
+        from ghostpkg.registries import RegistryError
+
+        monkeypatch.setattr(
+            "ghostpkg.scanner.fetch",
+            lambda name, ecosystem: (_ for _ in ()).throw(RegistryError("offline")),
+        )
+        path = tmp_path / "requirements.txt"
+        path.write_text("flask\n", encoding="utf-8")
+        main(["scan", str(path), "--no-cache"])
+        out = capsys.readouterr().out
+        assert "could not be checked" in out
+        assert "look fine" not in out

{ghostpkg-0.22.0 → ghostpkg-0.24.0}/tests/test_prose.py

@@ -198,3 +198,59 @@ class TestACommandInsideASentence:
 
     def test_a_span_with_a_backtick_pair_but_no_installer(self):
         assert self.names("The `requirements.txt` file lists them.") == []
+
+
+class TestAVersionInAnInstallCommand:
+    """`npm install react@18` and `pip install flask==3.0` both extracted
+    nothing at all. The `@` was read as a direct reference and the `==` made the
+    token stop matching the name pattern, so a pinned install command -- the
+    commonest form there is -- was silently skipped by the feature built to
+    read install commands.
+
+    Re-measured after the change, because widening this parser has cost false
+    positives before: 19 names across 22 real READMEs, none that fail to exist.
+    """
+
+    @pytest.mark.parametrize(
+        "line, name, specifier",
+        [
+            ("npm install react@18", "react", "18"),
+            ("npm i lodash@^4.17.21", "lodash", "^4.17.21"),
+            ("pnpm add vue@3", "vue", "3"),
+            ("yarn add @babel/core@7", "@babel/core", "7"),
+            ("bun add hono@4.0.0", "hono", "4.0.0"),
+            ("pip install flask==3.0", "flask", "3.0"),
+            ("pip install httpx>=0.27", "httpx", "0.27"),
+            ("pip install django~=5.0", "django", "5.0"),
+            ("uv add fastapi==0.115.0", "fastapi", "0.115.0"),
+            ("npm install react", "react", None),
+        ],
+    )
+    def test_the_name_and_the_version_are_both_read(self, line, name, specifier):
+        found = extract(line)
+        assert [(r.name, r.specifier) for r in found] == [(name, specifier)]
+
+    @pytest.mark.parametrize(
+        "line",
+        [
+            "npm install pkg@https://example.com/x.tgz",
+            "npm install pkg@file:../local",
+            "npm install pkg@acme/repo",
+            "npm install pkg@github:acme/repo",
+        ],
+    )
+    def test_a_stated_source_after_the_at_is_still_skipped(self, line):
+        """The `@` guard existed to catch these. Splitting the version off must
+        not let a direct reference through."""
+        assert extract(line) == []
+
+    def test_a_pinned_name_in_prose_keeps_its_line(self):
+        text = "intro\n\nRun `pip install ghost-thing==1.0` to begin.\n"
+        found = extract(text)
+        assert (found[0].name, found[0].specifier, found[0].line) == ("ghost-thing", "1.0", 3)
+
+    def test_the_sentence_still_ends_the_command(self):
+        """The 25% false-positive rate this parser once had came from running
+        past the full stop. Splitting versions must not reopen that."""
+        text = "Run `pip install httpx==0.27` and the command line client is optional."
+        assert [r.name for r in extract(text)] == ["httpx"]

ghostpkg-0.24.0/tests/test_trusted_scopes.py

@@ -0,0 +1,140 @@
+"""A scope the project already leans on is not a stranger.
+
+268 warnings across four popular repositories was the last barrier a reviewer
+named: almost all of them "published under a year ago" on the dozen platform
+binaries a compiled tool ships at once -- `@oxfmt/binding-darwin-arm64`,
+`@oxc-parser/binding-linux-riscv64-musl`.
+
+The justification is structural rather than statistical: **an npm scope is
+owned**. To publish `@oxfmt/anything` you must control `@oxfmt`. So a young
+package in a scope the scan already depends on three times is the same
+publisher shipping another build.
+
+Measured before it was built. Two members silences 41% of warnings, three
+silences 40%, six silences 39% -- it saturates at once, so the threshold sits
+where it is defensible rather than where it is largest. In practice: Sentry 114
+warnings to 22, Vue 15 to 5, and no change at all to the block count.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from ghostpkg.assess import TRUSTED_SCOPE_MEMBERS, Verdict, assess, trusted_scopes
+from ghostpkg.registries import PackageFacts
+
+
+def young(name, **kwargs):
+    base = dict(
+        name=name, ecosystem="npm", exists=True,
+        age_days=40, release_count=1, has_repo_url=False,
+    )
+    base.update(kwargs)
+    return PackageFacts(**base)
+
+
+class TestWhichScopesAreTrusted:
+    def test_a_scope_seen_often_enough(self):
+        names = [f"@oxfmt/binding-{i}" for i in range(TRUSTED_SCOPE_MEMBERS)]
+        assert trusted_scopes(names) == {"@oxfmt"}
+
+    def test_one_short_is_not_enough(self):
+        names = [f"@oxfmt/binding-{i}" for i in range(TRUSTED_SCOPE_MEMBERS - 1)]
+        assert trusted_scopes(names) == set()
+
+    def test_a_squatted_scope_appears_once(self):
+        """`@types-node` is not `@types/node`. A squatted scope shows up a
+        single time in a scan, which is exactly what the threshold separates."""
+        names = ["@types/node", "@types/react", "@types/jest", "@types-node/core"]
+        assert trusted_scopes(names) == {"@types"}
+
+    def test_unscoped_names_never_count(self):
+        assert trusted_scopes(["react", "vue", "lodash", "express"]) == set()
+
+    def test_scopes_are_counted_separately(self):
+        names = ["@a/one", "@a/two", "@a/three", "@b/one", "@b/two"]
+        assert trusted_scopes(names) == {"@a"}
+
+
+class TestWhatBeingTrustedChanges:
+    def test_the_soft_signals_are_dropped(self):
+        finding = assess(young("@oxfmt/binding-darwin-arm64"), known_scope=True)
+        assert finding.verdict is Verdict.OK
+        assert finding.reasons == []
+
+    def test_the_same_package_outside_a_trusted_scope_is_warned_about(self):
+        finding = assess(young("@oxfmt/binding-darwin-arm64"), known_scope=False)
+        assert finding.verdict is Verdict.WARN
+        assert len(finding.reasons) == 3
+
+    def test_a_missing_package_is_still_blocked(self):
+        """Nothing that blocks is affected. A trusted scope does not make a
+        name that does not exist acceptable."""
+        facts = PackageFacts(name="@oxfmt/ghost", ecosystem="npm", exists=False)
+        assert assess(facts, known_scope=True).verdict is Verdict.BLOCK
+
+    def test_a_bad_pin_is_still_blocked(self):
+        facts = young("@oxfmt/binding-x", versions=("1.0.0",), latest_version="1.0.0")
+        finding = assess(facts, specifier="99.9.9", known_scope=True)
+        assert finding.verdict is Verdict.BLOCK
+
+    def test_a_confiscated_name_is_still_blocked(self):
+        facts = young("@oxfmt/binding-x", security_hold=True)
+        assert assess(facts, known_scope=True).verdict is Verdict.BLOCK
+
+    def test_install_signals_still_block_a_young_package(self):
+        """`--deep` findings are evidence about the package itself, not a guess
+        from its age, so the scope says nothing about them."""
+        from ghostpkg.inspection import Signal
+
+        signals = [Signal(kind="pipe-to-shell", detail="curl | sh", where="package.json")]
+        finding = assess(young("@oxfmt/binding-x"), signals=signals, known_scope=True)
+        assert finding.verdict is Verdict.BLOCK
+
+
+class TestEndToEnd:
+    def test_a_family_of_binaries_produces_no_warnings(self, tmp_path, monkeypatch):
+        import json
+
+        from ghostpkg.cli import main
+
+        monkeypatch.setattr(
+            "ghostpkg.scanner.fetch",
+            lambda name, ecosystem: young(name, ecosystem=ecosystem),
+        )
+        deps = {f"@oxfmt/binding-{p}": "1.0.0" for p in ("darwin-arm64", "linux-x64", "win32-x64")}
+        (tmp_path / "package.json").write_text(
+            json.dumps({"name": "app", "dependencies": deps}), encoding="utf-8"
+        )
+        assert main(["scan", str(tmp_path / "package.json"), "--no-cache"]) == 0
+
+    def test_a_lone_scoped_newcomer_is_still_warned_about(self, tmp_path, monkeypatch, capsys):
+        import json
+
+        from ghostpkg.cli import main
+
+        monkeypatch.setattr(
+            "ghostpkg.scanner.fetch",
+            lambda name, ecosystem: young(name, ecosystem=ecosystem),
+        )
+        (tmp_path / "package.json").write_text(
+            json.dumps({"name": "app", "dependencies": {"@stranger/thing": "1.0.0"}}),
+            encoding="utf-8",
+        )
+        main(["scan", str(tmp_path / "package.json"), "--no-cache"])
+        assert "WARNING" in capsys.readouterr().out
+
+    def test_strict_still_fails_on_a_lone_newcomer(self, tmp_path, monkeypatch):
+        import json
+
+        from ghostpkg.cli import main
+
+        monkeypatch.setattr(
+            "ghostpkg.scanner.fetch",
+            lambda name, ecosystem: young(name, ecosystem=ecosystem),
+        )
+        (tmp_path / "package.json").write_text(
+            json.dumps({"name": "app", "dependencies": {"@stranger/thing": "1.0.0"}}),
+            encoding="utf-8",
+        )
+        assert main(["scan", str(tmp_path / "package.json"), "--strict", "--no-cache"]) == 1