ghostpkg 0.3.0__tar.gz → 0.5.0__tar.gz

{ghostpkg-0.3.0 → ghostpkg-0.5.0}/CHANGELOG.md

@@ -6,6 +6,68 @@ this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.5.0] - 2026-09-01
+
+### Added
+- **`--deep`: static inspection of install-time code.** This addresses the
+  project's main open problem ([#1]) — a hallucinated name an attacker has
+  *already registered*. Such a package exists, so the existence check passes,
+  and it is young with one release and no repository link, exactly like every
+  honest new package. Age cannot separate them; install-time behaviour can,
+  because a slopsquat has to run something when it is installed.
+- Signals reported: reading environment variables together with a network call,
+  a network request, a shell command, decoding a hidden blob, and executing code
+  that was just decoded or downloaded.
+
+### How the policy was decided
+The previous signal adopted on intuition — scoring packages by age — flagged
+100% of legitimate same-day publications. So this one was measured first:
+
+| Group | Flagged |
+|---|---|
+| 27 established legitimate packages | 0% |
+| 32 packages published to PyPI that day | 0% |
+| 6 known malicious install-script shapes | 6 of 6 |
+
+That is why a **young** package with install-time signals is **blocked** while
+age alone still only warns. An established package with the same signals is
+warned about, not blocked.
+
+The first pattern set was far looser and flagged 37% of established packages,
+mostly for reading environment variables — ordinary when inspecting build
+flags. It also scanned `conftest.py`, which runs during testing and never on
+install. Both were mistakes found by measuring rather than by reasoning.
+
+### Safety
+Archives are read in memory and never extracted to disk; nothing is executed,
+imported or compiled; downloads stop at 8 MB and members at 512 KB so a
+decompression bomb cannot exhaust memory; any failure to fetch or parse means
+"not inspected" rather than a pass.
+
+[#1]: https://github.com/M1rwana12/ghostpkg/issues/1
+
+## [0.4.0] - 2026-09-01
+
+### Added
+- **On-disk cache for registry lookups.** Scanning a 150-package manifest drops
+  from 4.7s to 0.4s on a warm cache. Previously every scan cost one request per
+  dependency, every run, which is slow in CI and rude to the registry.
+- `--no-cache` to bypass it, and `ghostpkg clear-cache` to delete it.
+- `GHOSTPKG_CACHE_DIR` to override the location. The default is
+  `%LOCALAPPDATA%\ghostpkg` on Windows, `~/Library/Caches/ghostpkg` on macOS and
+  `$XDG_CACHE_HOME/ghostpkg` elsewhere, worked out without a dependency.
+
+### Notes on the cache design
+- **Time-to-live depends on the answer, and "does not exist" is held for only an
+  hour.** A free name can be registered at any moment — that is the whole attack
+  — so a negative result must not be trusted for long. Young packages are held
+  six hours, established ones a day.
+- Cache failures are never fatal. A corrupt file, a wrong schema, a malformed
+  entry or an unwritable directory all degrade to no cache rather than breaking
+  a run. There are tests for each of those.
+- Written atomically via a temporary file and `os.replace`, once per run, so a
+  killed process cannot leave a half-written cache behind.
+
 ## [0.3.0] - 2026-09-01
 
 ### Fixed
@@ -87,7 +149,9 @@ First release.
 - No corpus of hallucinated package names is shipped, following the decision of
   the USENIX'25 authors not to publish theirs.
 
-[Unreleased]: https://github.com/M1rwana12/ghostpkg/compare/v0.3.0...HEAD
+[Unreleased]: https://github.com/M1rwana12/ghostpkg/compare/v0.5.0...HEAD
+[0.5.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.5.0
+[0.4.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.4.0
 [0.3.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.3.0
 [0.2.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.2.0
 [0.1.0]: https://github.com/M1rwana12/ghostpkg/releases/tag/v0.1.0

{ghostpkg-0.3.0 → ghostpkg-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ghostpkg
-Version: 0.3.0
+Version: 0.5.0
 Summary: Catch package names that do not exist before you install them
 Project-URL: Homepage, https://github.com/m1rwana12/ghostpkg
 Project-URL: Issues, https://github.com/m1rwana12/ghostpkg/issues
@@ -180,6 +180,8 @@ and report TOML keys as package names.
 | `--strict` | Promote warnings to blocks |
 | `--json` | Machine-readable output for scripts and CI |
 | `-q`, `--quiet` | Hide packages that passed |
+| `--no-cache` | Neither read nor write the cache |
+| `--deep` | Download recently published packages and statically inspect their install scripts |
 | `--version` | Print the version |
 
 ### Exit codes
@@ -190,6 +192,25 @@ and report TOML keys as package names.
 | `1` | At least one package blocked |
 | `2` | Usage error, unreadable manifest, or the registry was unreachable |
 
+### Caching
+
+Lookups are cached on disk, so re-scanning a 150-package manifest takes 0.4s
+instead of 4.7s.
+
+Time-to-live depends on the answer, and the negative case is the one that
+matters: **"does not exist" is held for only an hour**, because a free name can
+be registered at any moment and that is the entire attack. Young packages are
+held six hours, established ones a day.
+
+```bash
+ghostpkg scan requirements.txt --no-cache   # bypass it
+ghostpkg clear-cache                        # delete it
+GHOSTPKG_CACHE_DIR=/tmp/gp ghostpkg check x # move it
+```
+
+A corrupt, unreadable or unwritable cache degrades to no cache rather than
+breaking your run.
+
 ### JSON output
 
 ```console
@@ -328,11 +349,62 @@ roughly one round trip rather than one per dependency.
 
 ---
 
+## `--deep`: inspecting install scripts
+
+The existence check cannot see the dangerous case — a name an attacker has
+**already registered**. That package exists, so it passes; and it is young, with
+one release and no repository link, which describes every honest new package
+too. **Age cannot separate them.**
+
+Install-time behaviour can. A slopsquat has to run something when it is
+installed — that is the entire point of publishing it. An honest new library
+almost never does.
+
+```bash
+ghostpkg scan requirements.txt --deep
+```
+
+`--deep` downloads the archive **only for recently published packages**, reads
+only `setup.py` from it (or the install hooks out of `package.json`), and
+pattern-matches the text. **Nothing is ever executed.** Archive and member sizes
+are capped, so a decompression bomb cannot exhaust memory.
+
+| Signal | What it means |
+|---|---|
+| `exfiltration` | Reads environment variables *and* contacts the network |
+| `network` | Makes a network request during install |
+| `subprocess` | Runs a shell command during install |
+| `encoded-payload` | Decodes a hidden blob, or carries a large encoded string |
+| `dynamic-exec` | Executes code it just decoded or downloaded |
+
+### It was measured before it was allowed to block
+
+The last signal adopted on intuition — score packages by age — flagged 100% of
+legitimate same-day publications. So this one was measured first:
+
+| Group | Flagged |
+|---|---|
+| 27 established legitimate packages | **0%** |
+| 32 packages published to PyPI that day | **0%** |
+| 6 known malicious install-script shapes | **6 of 6** |
+
+That is why a **young** package with install-time signals is **blocked**, while
+age alone can only ever warn. An established package showing the same signals is
+warned about rather than blocked: old packages do sometimes build things at
+install time, and the sample behind that judgement is small.
+
+**Limits, stated plainly:** a squat that waits until import time rather than
+install time will not be caught, obfuscation beyond the listed patterns will not
+be caught, and packages published without an sdist cannot be inspected at all.
+
+---
+
 ## Comparison
 
 | | `ghostpkg` | SCA scanners (Snyk, Socket) | `pip install` alone |
 |---|:---:|:---:|:---:|
 | Catches a name that doesn't exist | ✅ **before install** | after install / in a PR | ❌ |
+| Inspects install scripts without running them | ✅ `--deep` | varies | ❌ |
 | Runs without an account | ✅ | ❌ | — |
 | Runtime dependencies | **0** | many | — |
 | Blocks legitimate new packages | ❌ **no** | varies | — |
@@ -348,16 +420,18 @@ earns its keep. Use both.
 ## Honest limitations
 
 > [!WARNING]
-> **The hard case is out of scope today.** A hallucinated name that an attacker has
-> **already registered** will pass the existence check. The warning signals are all
-> that stand between you and it, and they are advisory. Improving this is the main
-> open problem — see [issues](https://github.com/M1rwana12/ghostpkg/issues).
+> **The hard case is partly addressed by `--deep`, not solved.** A hallucinated
+> name an attacker has **already registered** passes the existence check.
+> `--deep` looks at install-time behaviour and catches the usual shapes, but an
+> attacker who does nothing during install will still get through. Discussion in
+> [#1](https://github.com/M1rwana12/ghostpkg/issues/1).
 
 - Typo detection compares against the 2,000 most-downloaded projects in each
   ecosystem, so a squat on a less popular package won't be flagged as a lookalike.
 - Names shorter than five characters are not compared at all: below that the name
   space is too dense for edit distance to mean anything.
-- Every check is a live registry request. There is no caching yet.
+- The cache lives on disk. `ghostpkg clear-cache` removes it and
+  `GHOSTPKG_CACHE_DIR` moves it.
 - Registry outages surface as exit code `2` rather than a silent pass — deliberately,
   but it does mean a flaky network fails your build.
 

{ghostpkg-0.3.0 → ghostpkg-0.5.0}/README.en.md

@@ -133,6 +133,8 @@ and report TOML keys as package names.
 | `--strict` | Promote warnings to blocks |
 | `--json` | Machine-readable output for scripts and CI |
 | `-q`, `--quiet` | Hide packages that passed |
+| `--no-cache` | Neither read nor write the cache |
+| `--deep` | Download recently published packages and statically inspect their install scripts |
 | `--version` | Print the version |
 
 ### Exit codes
@@ -143,6 +145,25 @@ and report TOML keys as package names.
 | `1` | At least one package blocked |
 | `2` | Usage error, unreadable manifest, or the registry was unreachable |
 
+### Caching
+
+Lookups are cached on disk, so re-scanning a 150-package manifest takes 0.4s
+instead of 4.7s.
+
+Time-to-live depends on the answer, and the negative case is the one that
+matters: **"does not exist" is held for only an hour**, because a free name can
+be registered at any moment and that is the entire attack. Young packages are
+held six hours, established ones a day.
+
+```bash
+ghostpkg scan requirements.txt --no-cache   # bypass it
+ghostpkg clear-cache                        # delete it
+GHOSTPKG_CACHE_DIR=/tmp/gp ghostpkg check x # move it
+```
+
+A corrupt, unreadable or unwritable cache degrades to no cache rather than
+breaking your run.
+
 ### JSON output
 
 ```console
@@ -281,11 +302,62 @@ roughly one round trip rather than one per dependency.
 
 ---
 
+## `--deep`: inspecting install scripts
+
+The existence check cannot see the dangerous case — a name an attacker has
+**already registered**. That package exists, so it passes; and it is young, with
+one release and no repository link, which describes every honest new package
+too. **Age cannot separate them.**
+
+Install-time behaviour can. A slopsquat has to run something when it is
+installed — that is the entire point of publishing it. An honest new library
+almost never does.
+
+```bash
+ghostpkg scan requirements.txt --deep
+```
+
+`--deep` downloads the archive **only for recently published packages**, reads
+only `setup.py` from it (or the install hooks out of `package.json`), and
+pattern-matches the text. **Nothing is ever executed.** Archive and member sizes
+are capped, so a decompression bomb cannot exhaust memory.
+
+| Signal | What it means |
+|---|---|
+| `exfiltration` | Reads environment variables *and* contacts the network |
+| `network` | Makes a network request during install |
+| `subprocess` | Runs a shell command during install |
+| `encoded-payload` | Decodes a hidden blob, or carries a large encoded string |
+| `dynamic-exec` | Executes code it just decoded or downloaded |
+
+### It was measured before it was allowed to block
+
+The last signal adopted on intuition — score packages by age — flagged 100% of
+legitimate same-day publications. So this one was measured first:
+
+| Group | Flagged |
+|---|---|
+| 27 established legitimate packages | **0%** |
+| 32 packages published to PyPI that day | **0%** |
+| 6 known malicious install-script shapes | **6 of 6** |
+
+That is why a **young** package with install-time signals is **blocked**, while
+age alone can only ever warn. An established package showing the same signals is
+warned about rather than blocked: old packages do sometimes build things at
+install time, and the sample behind that judgement is small.
+
+**Limits, stated plainly:** a squat that waits until import time rather than
+install time will not be caught, obfuscation beyond the listed patterns will not
+be caught, and packages published without an sdist cannot be inspected at all.
+
+---
+
 ## Comparison
 
 | | `ghostpkg` | SCA scanners (Snyk, Socket) | `pip install` alone |
 |---|:---:|:---:|:---:|
 | Catches a name that doesn't exist | ✅ **before install** | after install / in a PR | ❌ |
+| Inspects install scripts without running them | ✅ `--deep` | varies | ❌ |
 | Runs without an account | ✅ | ❌ | — |
 | Runtime dependencies | **0** | many | — |
 | Blocks legitimate new packages | ❌ **no** | varies | — |
@@ -301,16 +373,18 @@ earns its keep. Use both.
 ## Honest limitations
 
 > [!WARNING]
-> **The hard case is out of scope today.** A hallucinated name that an attacker has
-> **already registered** will pass the existence check. The warning signals are all
-> that stand between you and it, and they are advisory. Improving this is the main
-> open problem — see [issues](https://github.com/M1rwana12/ghostpkg/issues).
+> **The hard case is partly addressed by `--deep`, not solved.** A hallucinated
+> name an attacker has **already registered** passes the existence check.
+> `--deep` looks at install-time behaviour and catches the usual shapes, but an
+> attacker who does nothing during install will still get through. Discussion in
+> [#1](https://github.com/M1rwana12/ghostpkg/issues/1).
 
 - Typo detection compares against the 2,000 most-downloaded projects in each
   ecosystem, so a squat on a less popular package won't be flagged as a lookalike.
 - Names shorter than five characters are not compared at all: below that the name
   space is too dense for edit distance to mean anything.
-- Every check is a live registry request. There is no caching yet.
+- The cache lives on disk. `ghostpkg clear-cache` removes it and
+  `GHOSTPKG_CACHE_DIR` moves it.
 - Registry outages surface as exit code `2` rather than a silent pass — deliberately,
   but it does mean a flaky network fails your build.
 

{ghostpkg-0.3.0 → ghostpkg-0.5.0}/README.md

@@ -86,14 +86,24 @@ ghostpkg scan package.json
 
 # машинозчитуваний вивід
 ghostpkg check somepkg --json
+
+# видалити кеш
+ghostpkg clear-cache
 ```
 
+Результати запитів кешуються на диск: повторна перевірка манифесту зі 150
+залежностей займає 0,4 с замість 4,7 с. **Відповідь «пакета не існує»
+зберігається лише годину** — вільне ім'я можуть зареєструвати будь-якої
+миті, і саме в цьому вся атака.
+
 | Прапорець | Призначення |
 |---|---|
 | `-e`, `--ecosystem` | `pypi` (типово) або `npm` |
 | `--strict` | Підвищує попередження до блокувань |
 | `--json` | Вивід у JSON для скриптів і CI |
 | `-q`, `--quiet` | Ховає пакети, які пройшли перевірку |
+| `--no-cache` | Не читати й не писати кеш |
+| `--deep` | Завантажити свіжі пакети й статично перевірити їхні скрипти встановлення |
 
 `scan` розпізнає `requirements*.txt`, `pyproject.toml` (PEP 621 і Poetry)
 та `package.json`. Невідомий формат він **відхиляє з помилкою, а не вгадує**.
@@ -140,6 +150,42 @@ $ ghostpkg check react-router-dom-utils -e npm
 
 ---
 
+## `--deep`: перевірка скриптів встановлення
+
+Перевірка існування не бачить найнебезпечнішого випадку — імені, яке зловмисник
+**уже зареєстрував**. Такий пакет існує, тож проходить; він молодий, з одним
+релізом і без репозиторію — як і будь-який чесний новий пакет. **Вік їх не розрізняє.**
+
+Поведінка при встановленні — розрізняє. Слопсквот мусить щось виконати під час
+встановлення, інакше в ньому немає сенсу. Чесна нова бібліотека цього майже ніколи
+не робить.
+
+```bash
+ghostpkg scan requirements.txt --deep
+```
+
+`--deep` завантажує архів **лише для свіжих пакетів**, читає з нього тільки
+`setup.py` (або скрипти встановлення з `package.json`) і зіставляє текст із
+шаблонами. **Нічого не виконується.** Розмір архіва обмежений, тож бомба
+розпакування не з'їсть пам'ять.
+
+Що вважається сигналом: читання змінних середовища разом із мережевим запитом,
+мережевий запит, запуск оболонки, розкодування прихованого блоба, виконання
+щойно розкодованого коду.
+
+**Виміряно перед тим, як вмикати блокування:**
+
+| Група | Позначено |
+|---|---|
+| 27 усталених легітимних пакетів | **0 %** |
+| 32 пакети, опубліковані того ж дня | **0 %** |
+| 6 відомих шкідливих шаблонів | **6 з 6** |
+
+Для порівняння: вік позначав **100 %** свіжих легітимних пакетів. Саме тому
+молодий пакет із такими сигналами **блокується**, а вік — лише попереджає.
+
+---
+
 ## Порівняння
 
 | | `ghostpkg` | SCA-сканери (Snyk, Socket) | Просто `pip install` |
@@ -158,16 +204,18 @@ $ ghostpkg check react-router-dom-utils -e npm
 ## Чесні обмеження
 
 > [!WARNING]
-> **Складний випадок поки поза межами інструменту.** Вигадану назву, яку зловмисник
-> **уже зареєстрував**, перевірка існування пропустить. Між вами й нею стоять лише
-> попередження, і вони дорадчі. Це головна відкрита проблема —
-> див. [issues](https://github.com/M1rwana12/ghostpkg/issues).
+> **Складний випадок частково закрито прапорцем `--deep`, але не повністю.**
+> Вигадану назву, яку зловмисник **уже зареєстрував**, перевірка існування
+> пропустить. `--deep` дивиться на поведінку при встановленні й ловить типові
+> шаблони, але зловмисник, який нічого не робить під час встановлення, пройде.
+> Обговорення — [#1](https://github.com/M1rwana12/ghostpkg/issues/1).
 
 - Виявлення опечаток порівнює з 2 000 найпопулярніших проєктів кожної екосистеми,
   тому підробка під менш популярний пакет як схожа назва не позначиться.
 - Імена коротші за 5 символів не порівнюються: там простір назв надто щільний,
   щоб відстань редагування щось означала.
-- Кожна перевірка — це живий запит до реєстру. Кешування ще немає.
+- Кеш зберігається локально. `ghostpkg clear-cache` видаляє його,
+  `GHOSTPKG_CACHE_DIR` змінює розташування.
 
 ---
 

{ghostpkg-0.3.0 → ghostpkg-0.5.0}/SECURITY.md

@@ -20,13 +20,17 @@ what it does and does not protect against.
 
 ### What it does not catch
 
-- **A hallucinated name an attacker has already registered.** The existence check
-  passes. Only the advisory warnings stand between the user and it. This is the
-  main open problem and it is stated plainly in the README rather than hidden.
+- **A hallucinated name an attacker has already registered, when `--deep` is
+  off.** The existence check passes and only advisory warnings remain. With
+  `--deep`, install-time code is statically inspected and the usual malicious
+  shapes are caught, which measured 0 false positives across 27 established and
+  32 same-day packages while catching all 6 test shapes.
+- **Even with `--deep`:** a squat whose payload runs at *import* time rather than
+  install time, obfuscation beyond the documented patterns, and any package
+  published without an sdist, which cannot be inspected.
 - Malicious code in a package that is otherwise legitimate and established.
 - Compromise of an existing maintainer account.
-- Anything at install time. `ghostpkg` inspects registry metadata; it never
-  downloads, unpacks or executes a package.
+- Malicious behaviour at *runtime*. `--deep` reads install-time code only.
 
 ### Failure mode
 
@@ -34,9 +38,20 @@ If a registry is unreachable, `ghostpkg` exits with code `2` rather than passing
 silently. A network failure will fail your build. That is deliberate: a security
 check that quietly succeeds when it could not run is worse than no check.
 
+### How `--deep` handles untrusted archives
+
+- Archives are read **in memory**, never extracted to disk, so a path-traversal
+  entry has nothing to write to.
+- **Nothing is executed, imported or compiled.** Only named install-time files
+  are read, and only as text.
+- Downloads stop at 8 MB and individual members at 512 KB, so a decompression
+  bomb cannot exhaust memory.
+- Any failure to download or parse means "not inspected", never a pass.
+
 ### Trust boundaries
 
-- Requests go only to `pypi.org` and `registry.npmjs.org` over HTTPS.
+- Requests go only to `pypi.org`, `files.pythonhosted.org` and
+  `registry.npmjs.org` over HTTPS.
 - No telemetry, no analytics, no phoning home.
 - No runtime dependencies, so the tool's own supply chain is the Python standard
   library.

{ghostpkg-0.3.0 → ghostpkg-0.5.0}/ghostpkg/__init__.py

@@ -1,6 +1,6 @@
 """ghostpkg -- catch package names that do not exist before you install them."""
 
-__version__ = "0.3.0"
+__version__ = "0.5.0"
 
 from .assess import Finding, Verdict, assess
 from .registries import PackageFacts, RegistryError, fetch

{ghostpkg-0.3.0 → ghostpkg-0.5.0}/ghostpkg/assess.py

@@ -134,7 +134,26 @@ def nearest_popular(name: str, ecosystem: str = "pypi") -> tuple[str, int] | Non
     return best
 
 
-def assess(facts: PackageFacts, strict: bool = False) -> Finding:
+def assess(
+    facts: PackageFacts,
+    strict: bool = False,
+    signals: "list | None" = None,
+) -> Finding:
+    """Turn registry facts, and optionally --deep install-script signals, into
+    a verdict.
+
+    Install-time signals are treated differently from every other soft signal,
+    and the difference is measured rather than assumed. Age flags 100% of
+    legitimate same-day publications, so it can only ever warn. Install-time
+    behaviour flagged 0 of 27 established and 0 of 32 brand-new real packages
+    while catching all six known malicious shapes, so a *young* package that
+    reaches for the network, a subprocess or a decoded payload during install
+    is specific enough to block.
+
+    An established package doing the same is only warned about: legitimate
+    old packages do sometimes build things at install time, and the sample
+    behind that judgement is small.
+    """
     if not facts.exists:
         return Finding(
             name=facts.name,
@@ -169,7 +188,12 @@ def assess(facts: PackageFacts, strict: bool = False) -> Finding:
     if is_young and not facts.has_repo_url:
         reasons.append("no repository or homepage link")
 
-    if not reasons:
+    install_reasons = [str(signal) for signal in (signals or [])]
+    reasons.extend(install_reasons)
+
+    if install_reasons and is_young:
+        verdict = Verdict.BLOCK
+    elif not reasons:
         verdict = Verdict.OK
     elif strict:
         verdict = Verdict.BLOCK