diff-sentry 0.12.0__tar.gz → 0.13.0__tar.gz

{diff_sentry-0.12.0 → diff_sentry-0.13.0}/.claude/rules/monitoring.md

@@ -54,6 +54,13 @@ paths:
 - **The previous release** is the highest version below the new one with a file that is not yanked; a pre-release is compared only when the new one is a pre-release too. Members are compared by content across all artifacts of a kind, so a payload in one platform's wheel shows.
 - **State is per file.** First sight of a package (even an empty one) records every file as the baseline. After that, each file not yet recorded is judged with the other new files of its release, against the previous release, so a wheel uploaded after its release was judged, or added to an old release, is judged too. A file is recorded only after its comparison is screened, and the index's ETag only after every new file of it is: a watcher stopped mid-comparison judges the same files on its next pass. A failed comparison is retried on every pass (the ETag is not sent while one is pending), superseded when the files to judge change, and kept as `error` after five, with an alert that it was not compared. A `--check` runs under `...-check` and never stands in for the watcher's judgment. An alert the SIEM did not accept is re-sent from `screened/` a few per poll.
 
+## npm watch (`npm.py`, `release-watch --npm`)
+
+- **A version is a release.** `NpmWatcher` is `ReleaseWatcher` with `registry = "npm"` and its own `npm.db`: the same per-file state, retries, give-up and alert re-sending, with a version's tarball as its one file (kind `npm`, labelled `npm/<path>`, the `package/` top directory dropped). `release_event(registry="npm")` orders versions by semver (`semver_key`: a release after its pre-releases, numeric identifiers before others), keeps the name as written (`@scope/name`; run id `npm___scope_name-<version>`), skips the tag comparison and the wheel facts, and adds `npm_facts`.
+- **Read from the registry only.** The abbreviated packument (`application/vnd.npm.install-v1+json`, ETag-polled) from `registry.npmjs.org`; a version whose tarball URL is on another host is left out (`parse_packument`). Each tarball is checked against the registry's sha512 `integrity`, a mismatch a note.
+- **Publishers come from the registry's SLSA provenance** (`npm_publishers`): the `https://slsa.dev/provenance/v1` statement's `externalParameters.workflow` repository and path, written as `releases.publisher_identity` would (`github:owner/name:path::`), so `scan_release`'s publisher rules apply unchanged. The statement's subject must name the tarball's own sha512 (and a version the registry gives no sha512 for cannot be bound), or it is a note and no publisher. A 404 is no provenance; another status is retried like a download.
+- **Install scripts** (`npm_facts`): never the packument's `hasInstallScript`, and never through the screen's text budget, whose bounds and decoding fail open for a manifest. `tarball_manifests` reads the tarball itself as npm extracts it: the first path component dropped (a member at the root is not installed), the last entry of a path standing, the bytes decoded as UTF-8 with a BOM dropped and invalid bytes replaced, every `package.json` of the root and of a dependency under `node_modules` at any depth (npm runs a bundled dependency's install scripts). A manifest that is missing, not a regular file, past 16 MiB, not JSON (a `RecursionError` included) or not an object is a note (`release-not-fully-read`), never a package without scripts; the previous version's unread manifests count as none. The scripts are those an install from the registry runs (`preinstall`, `install`, `postinstall`; `prepare` runs only for a git or local install) and the implied `node-gyp rebuild` of a `binding.gyp` without either install script, keyed by package directory. Added is `npm-install-script-added` (high), a changed command `npm-install-script-changed` (medium). A dependency of any manifest (`dependencies`, `optionalDependencies`, `peerDependencies`) whose spec holds `:`, `/` or `\\`, or starts with `.` or `~/` (a git repository, a URL, a path, an `npm:` alias; a semver range or a dist-tag never does), new or with another spec than before, is `npm-dependency-source` (high). Manifest and `binding.gyp` names match in any case, as a case-insensitive file system finds them; two manifests whose paths differ only in case are each read and are a note, since which one npm reads depends on the file system. A path with `..` is not read, as npm refuses it. A new name in the root's `dependencies`, `optionalDependencies` or `peerDependencies` is `release-new-dependency` (low). A dependency newly taken from the registry in any manifest, keyed `<dir> <name>` (`_registry_dependencies`; a name whose spec moves onto the registry counts), is looked up (`dependency_facts`, at most 50): any version with `hasInstallScript` in its abbreviated document is `npm-dependency-runs-at-install` (high), and a package whose full document's `time.created` is under 30 days before the version's publish time (`published_at`, the watched package's `time[version]`; the clock only when that cannot be read), or a 404, is `npm-dependency-young` (high). A full document past 64 MiB is a note, never an established package, since an author pads versions as easily as they publish them; another failure is retried like a download.
+
 ## Image watch (`images.py`, `release-watch --images`)
 
 - **Provenance only, never a layer.** `image_provenance` reads an image's index, each BuildKit attestation manifest and its SLSA provenance layer; `provenance_facts` keeps the source repository (`vcs:source`; `owner/name` only for a URL whose host is exactly github.com, any other source as written, so a lookalike is only judged between GitHub repositories), revision, building repository, workflow without its ref, ref and ref protection. An attestation counts only when its `vnd.docker.reference.digest` is a platform image of the same index and the provenance's `subject` names that image (BuildKit always writes one); a platform image no attestation names is a note. A layer's content is a known gap.

{diff_sentry-0.12.0 → diff_sentry-0.13.0}/.env.example

@@ -79,6 +79,7 @@ DS_API_KEY=sk-...
 # DS_RELEASE_PACKAGES=litellm,requests     # release-watch: PyPI projects (or --packages / --packages-file)
 # DS_RELEASE_INTERVAL=300                  # release-watch: seconds between polls of each project's index
 # DS_RELEASE_MAX_ARTIFACT_MB=256           # release-watch: a larger wheel or sdist is not compared, which alerts
+# DS_RELEASE_NPM=@scope/name,left-pad    # release-watch: npm packages (or --npm)
 # DS_RELEASE_SOURCES=litellm=BerriAI/litellm  # release-watch: the GitHub repository each package is built from, compared by tag (or --sources)
 # DS_RELEASE_IMAGES=ghcr.io/owner/name     # release-watch: container images compared by build provenance (or --images)
 # DS_RELEASE_IMAGE_TAGS=latest             # release-watch: floating tags whose moves are judged (or --image-tags)

{diff_sentry-0.12.0 → diff_sentry-0.13.0}/AGENTS.md

@@ -31,10 +31,10 @@ rlm-harness is an exact PyPI pin (`rlm-harness==X.Y.Z` in `pyproject.toml`, lock
 - **The budget is hard: `max_retries=1`, no whole-RLM retry inside a run.** One change = one trajectory, so the trace stays valid training data. (`watch` may run a change again, but only when the previous attempt produced no result at all, an infrastructure failure; `cli.run` resets that run id's trace first, so exactly one trajectory per run id survives.) A failed run is infra before it is a schema bug; see `.claude/rules/detect-setup.md` for how the failure names itself.
 - **A second model-judgement is a TOOL, never the sub-LM.** `deep_classify` (on rlm-harness's `make_model_tool`) is the swappable second-stage seam the planner chooses to call, so the decision is a `tool_call` in the trajectory. The analyst intercept (`intercept_sub_lm`) is tracing-only, zero transforms. Swapping the backend touches only `classify_backend` / `_selfclassify_chat`. The planner passes distilled findings, never the whole diff.
 - **Sandboxed code builds no sub-agents.** In dspy 3.4.1 and later, sandboxed code can construct dspy programs that run on the host, call the analyst, can drive the task's tools, and record as one `sub_call` with no inner turns. The kit's default allows it, so `setup` passes `sandbox_sub_agents=False` explicitly, and production never injects an interpreter (which would bypass it); `test_program_offers_no_sub_agents` holds the behaviour.
-- **GitHub ingest, monitoring, and SIEM emission are host-side plumbing, never planner tools.** `ingest.py` shells out to `gh` host-side (transport injectable); `emit.emit_signal` POSTs and never raises; `watch`, `actors` and `snapshot` poll, remember, and archive outside the trajectory, `releases` reads PyPI's own hosts, and the refs and trees of a package's GitHub repository, as data that is never installed, and `images` reads the GHCR and Docker Hub APIs only (manifests and provenance, never a layer). Only choices the policy makes are tools; delivering a finished result is plumbing. What the store learns reaches the signal only as provenance facts through `scan_provenance`, never as a second derivation (see `.claude/rules/monitoring.md`).
+- **GitHub ingest, monitoring, and SIEM emission are host-side plumbing, never planner tools.** `ingest.py` shells out to `gh` host-side (transport injectable); `emit.emit_signal` POSTs and never raises; `watch`, `actors` and `snapshot` poll, remember, and archive outside the trajectory, `releases` and `npm` read PyPI's and npm's own hosts, and the refs and trees of a package's GitHub repository, as data that is never installed, and `images` reads the GHCR and Docker Hub APIs only (manifests and provenance, never a layer). Only choices the policy makes are tools; delivering a finished result is plumbing. What the store learns reaches the signal only as provenance facts through `scan_provenance`, never as a second derivation (see `.claude/rules/monitoring.md`).
 - **This is a ROLLOUT source: trajectories, not reward.** `rl_export` passes `reward=None`; labels read the assembled verdict and metrics are objective effort counters. `deep_classify` tool_calls train the classifier; every other action is the planner's. A metric keeps ONE meaning across rlm-harness versions (the corpus mixes them; `run_start.payload.rlm_harness` names the writer): a quantity a newer kit makes observable gets a NEW key that is None where the trace cannot say (e.g. `analyst_failures`). The ATLAS rubric and the eval member stay reward-free too; see `.claude/rules/rubric-and-eval.md`.
 - **Attack knowledge lives in `diff_sentry/skills/`, not the prompt.** The skill catalog is injected (`load_skills_as_tools(discovery="inject")`) and `read_skill(name)` pulls a body on demand. Fix a wrong convention by editing or adding a skill; `detect.INSTRUCTIONS` holds only identity, the MISSION frame, the tool cost model, the triage loop, and output-gating rules. Skills ship via `packages = ["diff_sentry"]`; a force-include duplicates them and breaks the build.
-- **Keep the dspy-free modules dspy-free.** `config`, `schema`, `normalize`, `indicators`, `shelllex`, `linewide`, `assemble`, `response`, `emit`, `ingest`, `hosts`, `dbutil`, `actors`, `snapshot`, `watch`, `releases`, `images`, `rl_export`, and `rubric` must not import dspy at module top, and `import diff_sentry` must not import dspy (`ClassifyChange` / `setup` / `run` / `screen` / `detect_from_event` are lazy PEP 562 re-exports).
+- **Keep the dspy-free modules dspy-free.** `config`, `schema`, `normalize`, `indicators`, `shelllex`, `linewide`, `assemble`, `response`, `emit`, `ingest`, `hosts`, `dbutil`, `actors`, `snapshot`, `watch`, `releases`, `npm`, `images`, `rl_export`, and `rubric` must not import dspy at module top, and `import diff_sentry` must not import dspy (`ClassifyChange` / `setup` / `run` / `screen` / `detect_from_event` are lazy PEP 562 re-exports).
 - **The trace self-describes the run.** `run_start` meta carries the normalized event, instructions, source echo, `baseline_indicators`, `emit_on`, role→model names, budgets, and the rubric skeleton, so an offline re-render/export re-derives the same `signal` (reading `emit_on` from meta, never current config). Any new per-run config that affects read-time derivation must ride in meta.
 
 ## Topic guides
@@ -54,5 +54,5 @@ Claude Code loads each rule automatically when you touch a matching file; other
 
 - rlm-harness owns the generic cores: `make_model_tool` (chat → retry → validate → circuit-break) under `deep_classify`, and the SSRF primitives (`is_safe_url`, `resolved_host_is_safe`, `parse_cidrs`) under `fetch_tool`, which adds the GitHub allowlist, the httpx provider, and tracing.
 - When this consumer needs a workaround, fix the reusable gap in rlm-harness generically; never special-case diff-sentry there. Consumer values (`DS_*` roles, the verdict schema, indicator rules, the SIEM payload shape) stay here.
-- The deterministic tier ships as `cli.screen`, `watch` and `release-watch` (host-side, in front of `cli.run`). Next increments: an image's new layers by content, and npm releases beside PyPI's; an optional small single-shot model call inside that tier; a GH Archive backfill for account history older than the Events API's 30 days; per-repository workflow awareness (which repos run `issue_comment` or `pull_request_target` workflows) to weight the slash-command rule.
+- The deterministic tier ships as `cli.screen`, `watch` and `release-watch` (host-side, in front of `cli.run`). Next increments: an image's new layers by content; an optional small single-shot model call inside that tier; a GH Archive backfill for account history older than the Events API's 30 days; per-repository workflow awareness (which repos run `issue_comment` or `pull_request_target` workflows) to weight the slash-command rule.
 - Two workspace members, never in the `diff_sentry` wheel: `studio/` (`diff-sentry-studio`, the detection console, behind its `live` extra) and `eval/` (`diff-sentry-eval`, the reward-free scorecard). Both read the trace / `DetectionResponse` contract through the public surface only. Keep the root wheel at `packages = ["diff_sentry"]` so `uv build` never sweeps them in.

{diff_sentry-0.12.0 → diff_sentry-0.13.0}/CHANGELOG.md

@@ -2,6 +2,19 @@
 
 All notable changes to diff-sentry, a BewAIre-style detector that classifies one GitHub change (PR, issue or push) for malicious intent on [`rlm-harness`](https://github.com/qazbnm456/rlm-harness).
 
+## 0.13.0
+
+Upgrade from 0.12.0: `release-watch` can watch npm packages. The GitHub Action's scan is unchanged, nothing needs configuring, and `rlm-harness==1.24.0` is unchanged.
+
+### Added
+- **`release-watch --npm` watches npm packages** (`DS_RELEASE_NPM`, or `--check npm:NAME VERSION` for one version). Each new version is compared with the version before it in semver order, as a PyPI release is: the tarball is read from `registry.npmjs.org` as data, checked against the registry's sha512, and every new or changed file is screened by the same rules. The npm worms of 2025 ran their payload from an install script, and a compromised version can also depend on a package published hours before, so it adds what only npm shows:
+  - `npm-install-script-added` (`high`): a `preinstall`, `install` or `postinstall` script, or the `node-gyp rebuild` a `binding.gyp` implies, that the previous version did not run, in the package or in a dependency it bundles. A script that changes command is `medium`.
+  - `npm-dependency-source` (`high`): a dependency newly installed from a git repository, a URL, a path or an `npm:` alias.
+  - `npm-dependency-runs-at-install` and `npm-dependency-young` (`high`): a dependency newly taken from the registry whose versions run a script at install, or that was created within 30 days of the version or is not on the registry.
+  - The publisher checks PyPI releases have, from the SLSA provenance the registry holds, which must name the tarball's own digest.
+
+  The manifests are read as npm extracts them, and one that cannot be read alerts. See [npm packages](docs/release-watch.md#npm-packages).
+
 ## 0.12.0
 
 Upgrade from 0.11.0: `release-watch` can compare each release with its git tag, and `rlm-harness` moves to 1.24.0. The GitHub Action's scan is unchanged, and nothing needs configuring.

{diff_sentry-0.12.0 → diff_sentry-0.13.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: diff-sentry
-Version: 0.12.0
+Version: 0.13.0
 Summary: Catch malicious pull requests before they merge: scan a diff for supply-chain attack patterns with deterministic evidence a model cannot suppress.
 Author: Boik Su
 License-Expression: MIT
@@ -43,7 +43,7 @@ jobs:
       - uses: actions/checkout@v7
         with:
           fetch-depth: 0          # the scan diffs against the PR base
-      - uses: qazbnm456/diff-sentry@v0.12.0
+      - uses: qazbnm456/diff-sentry@v0.13.0
 ```
 
 That is the whole setup. The job fails when an indicator at or above `high` fires. Add `push` to `on:` to also scan commits pushed straight to a branch. A push that bypasses branch protection can also delete this workflow, so `watch` and `release-watch` read the repository and its releases from outside.
@@ -96,7 +96,7 @@ If you operate the infrastructure, or maintain a project where a wrong call is e
 - **Deep classification**: a reasoning model reads a PR, issue or push in a sandbox and returns a verdict next to the evidence. See [How it works](docs/how-it-works.md).
 - **The studio**: a local console that streams a run as it happens. See [`studio/README.md`](studio/README.md).
 - **Watch**: monitors a list of repositories, alerts the moment the rules fire, and remembers the accounts and payload hosts it has seen. See [Watching a watchlist](docs/watch.md).
-- **Release watch**: compares each new release of the PyPI packages you depend on with the release before it, and alerts when the new wheels carry a startup hook or a packed loader that the source never showed. See [Watching releases](docs/release-watch.md).
+- **Release watch**: compares each new release of the PyPI and npm packages you depend on with the release before it, and alerts when it carries a startup hook, a packed loader or a new install script, or was published without its usual provenance. See [Watching releases](docs/release-watch.md).
 - **Trajectories for training**: every run is recorded, and `export` turns the traces into a reward-free dataset for fine-tuning a smaller planner. See [The full pipeline](docs/how-it-works.md#the-full-pipeline).
 
 None of this is needed for the Action, and none of it sends your code anywhere you did not configure yourself.
@@ -109,7 +109,7 @@ A strong prompt injection can skew a model's verdict, though never the evidence.
 
 - [How it works](docs/how-it-works.md): severity tuning, the evidence union, the full pipeline, running it locally, and the incidents the tests reconstruct.
 - [Watching a watchlist](docs/watch.md): the monitor, its alerts, its archive, and what the account dossier holds.
-- [Watching releases](docs/release-watch.md): comparing each new PyPI release with the one before it.
+- [Watching releases](docs/release-watch.md): comparing each new PyPI or npm release, and each container image, with what came before.
 - [Honest limits](docs/limits.md): what the rules and the pipeline do not see.
 - [Development](docs/development.md): the test suites and CI.
 - [CHANGELOG](CHANGELOG.md)

{diff_sentry-0.12.0 → diff_sentry-0.13.0}/README.md

@@ -21,7 +21,7 @@ jobs:
       - uses: actions/checkout@v7
         with:
           fetch-depth: 0          # the scan diffs against the PR base
-      - uses: qazbnm456/diff-sentry@v0.12.0
+      - uses: qazbnm456/diff-sentry@v0.13.0
 ```
 
 That is the whole setup. The job fails when an indicator at or above `high` fires. Add `push` to `on:` to also scan commits pushed straight to a branch. A push that bypasses branch protection can also delete this workflow, so `watch` and `release-watch` read the repository and its releases from outside.
@@ -74,7 +74,7 @@ If you operate the infrastructure, or maintain a project where a wrong call is e
 - **Deep classification**: a reasoning model reads a PR, issue or push in a sandbox and returns a verdict next to the evidence. See [How it works](docs/how-it-works.md).
 - **The studio**: a local console that streams a run as it happens. See [`studio/README.md`](studio/README.md).
 - **Watch**: monitors a list of repositories, alerts the moment the rules fire, and remembers the accounts and payload hosts it has seen. See [Watching a watchlist](docs/watch.md).
-- **Release watch**: compares each new release of the PyPI packages you depend on with the release before it, and alerts when the new wheels carry a startup hook or a packed loader that the source never showed. See [Watching releases](docs/release-watch.md).
+- **Release watch**: compares each new release of the PyPI and npm packages you depend on with the release before it, and alerts when it carries a startup hook, a packed loader or a new install script, or was published without its usual provenance. See [Watching releases](docs/release-watch.md).
 - **Trajectories for training**: every run is recorded, and `export` turns the traces into a reward-free dataset for fine-tuning a smaller planner. See [The full pipeline](docs/how-it-works.md#the-full-pipeline).
 
 None of this is needed for the Action, and none of it sends your code anywhere you did not configure yourself.
@@ -87,7 +87,7 @@ A strong prompt injection can skew a model's verdict, though never the evidence.
 
 - [How it works](docs/how-it-works.md): severity tuning, the evidence union, the full pipeline, running it locally, and the incidents the tests reconstruct.
 - [Watching a watchlist](docs/watch.md): the monitor, its alerts, its archive, and what the account dossier holds.
-- [Watching releases](docs/release-watch.md): comparing each new PyPI release with the one before it.
+- [Watching releases](docs/release-watch.md): comparing each new PyPI or npm release, and each container image, with what came before.
 - [Honest limits](docs/limits.md): what the rules and the pipeline do not see.
 - [Development](docs/development.md): the test suites and CI.
 - [CHANGELOG](CHANGELOG.md)

{diff_sentry-0.12.0 → diff_sentry-0.13.0}/action.yml

@@ -40,7 +40,7 @@ inputs:
       that PR rather than by the ones already on main.
     required: false
     # KEEP IN STEP WITH [project].version. release.yml fails the build if this drifts from the tag being released, because a stale default here would silently run an old scanner under a new action tag.
-    default: '0.12.0'
+    default: '0.13.0'
 
 outputs:
   failed:

{diff_sentry-0.12.0 → diff_sentry-0.13.0}/diff_sentry/__init__.py

@@ -72,7 +72,7 @@ __all__ = [
     "screen",
 ]
 
-__version__ = "0.12.0"
+__version__ = "0.13.0"
 
 
 def __getattr__(name: str):  # PEP 562: defer the dspy import to first use

{diff_sentry-0.12.0 → diff_sentry-0.13.0}/diff_sentry/cli.py

@@ -337,6 +337,7 @@ def _cmd_release_watch(args) -> int:
     import signal
 
     from .images import ImageWatcher
+    from .npm import NpmWatcher, parse_npm_packages
     from .releases import ReleaseWatcher, parse_packages, parse_sources
 
     logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
@@ -349,8 +350,11 @@ def _cmd_release_watch(args) -> int:
             with open(args.packages_file, encoding="utf-8") as fh:
                 texts.append(fh.read())
         packages = parse_packages(texts)
+        npm = parse_npm_packages([*args.npm, os.getenv("DS_RELEASE_NPM", "")])
+        check_npm = bool(args.check) and args.check[0].startswith("npm:")
         if args.check:
-            packages = parse_packages([args.check[0]])
+            packages = [] if check_npm else parse_packages([args.check[0]])
+            npm = parse_npm_packages([args.check[0][4:]]) if check_npm else []
         images = [i for text in [*args.images, os.getenv("DS_RELEASE_IMAGES", "")] for i in text.replace(",", " ").split()]
         floating = tuple(args.image_tags or os.getenv("DS_RELEASE_IMAGE_TAGS", "latest").replace(",", " ").split())
         image_watcher = ImageWatcher([] if args.check else images, state_dir=args.state, out_dir=args.out,
@@ -362,20 +366,26 @@ def _cmd_release_watch(args) -> int:
         except BaseException:
             image_watcher.close()
             raise
+        try:
+            npm_watcher = NpmWatcher(npm, state_dir=args.state, out_dir=args.out, emit=not args.no_emit)
+        except BaseException:
+            image_watcher.close()
+            watcher.close()
+            raise
     except (OSError, ValueError, RuntimeError) as exc:
         print(f"release-watch: {exc}", file=sys.stderr)
         return 2
     try:
         if args.check:
-            result = watcher.check(packages[0], args.check[1])
+            result = npm_watcher.check(npm[0], args.check[1]) if check_npm else watcher.check(packages[0], args.check[1])
             print(json.dumps(result, indent=2, default=str))
             return 1 if "error" in result else 0
-        if not packages and not images:
-            print("release-watch: nothing to watch (--packages, --packages-file, --images, or DS_RELEASE_PACKAGES / "
-                  "DS_RELEASE_IMAGES)", file=sys.stderr)
+        if not packages and not npm and not images:
+            print("release-watch: nothing to watch (--packages, --packages-file, --npm, --images, or DS_RELEASE_PACKAGES / "
+                  "DS_RELEASE_NPM / DS_RELEASE_IMAGES)", file=sys.stderr)
             return 2
         if args.once:
-            for result in watcher.poll_once() + image_watcher.poll_once():
+            for result in watcher.poll_once() + npm_watcher.poll_once() + image_watcher.poll_once():
                 print(json.dumps(result, default=str))
             return 0
         stopping = []
@@ -384,6 +394,7 @@ def _cmd_release_watch(args) -> int:
             while not stopping:
                 started = time.monotonic()
                 watcher.poll_once()
+                npm_watcher.poll_once()
                 image_watcher.poll_once()
                 while not stopping and time.monotonic() - started < args.interval:
                     time.sleep(min(1.0, args.interval))
@@ -392,6 +403,7 @@ def _cmd_release_watch(args) -> int:
         return 0
     finally:
         watcher.close()
+        npm_watcher.close()
         image_watcher.close()
 
 
@@ -657,6 +669,7 @@ def build_parser() -> argparse.ArgumentParser:
                         help="compare each new release of watched PyPI packages with the one before it (no model)")
     rw.add_argument("--packages", nargs="+", default=[], help="PyPI project names (or set DS_RELEASE_PACKAGES)")
     rw.add_argument("--packages-file", help="a file of project names, one per line (# comments allowed)")
+    rw.add_argument("--npm", nargs="+", default=[], help="npm package names, @scope/name included (or set DS_RELEASE_NPM)")
     rw.add_argument("--sources", nargs="+", default=[], metavar="PACKAGE=OWNER/NAME",
                     help="the GitHub repository each package is built from, to compare each release with its tag "
                          "(or set DS_RELEASE_SOURCES; default: the repository of the previous release's Trusted Publisher)")
@@ -673,7 +686,8 @@ def build_parser() -> argparse.ArgumentParser:
     rw.add_argument("--no-emit", action="store_true", help="do not emit SIEM signals")
     rw.add_argument("--once", action="store_true", help="one poll, then exit")
     rw.add_argument("--check", nargs=2, metavar=("PACKAGE", "VERSION"),
-                    help="judge one release against the one before it now, print the result, and exit")
+                    help="judge one release against the one before it now, print the result, and exit "
+                         "(npm:NAME for an npm package)")
     rw.set_defaults(func=_cmd_release_watch)
 
     ac = sub.add_parser("actor", help="the dossier of an account the watcher has seen (offline)")

diff_sentry-0.13.0/diff_sentry/npm.py

@@ -0,0 +1,337 @@
+"""npm watch: each new version of a watched npm package, compared with the version before it.
+
+The npm worms of 2025 published versions of popular packages with stolen tokens, each with a `preinstall` or `postinstall` script that runs the payload on every install; a version published with a stolen token also lacks the provenance a package's own release workflow attaches. So a version is a release like a PyPI one (`releases.release_event` with `registry="npm"`): its one tarball, read as DATA and never installed, against the previous version's, every new or changed file screened by the same rules, and the facts only npm shows added here. Every version's tarball is checked against the registry's sha512 `integrity`; publishers are read from the registry's SLSA provenance, bound to the tarball by its digest.
+
+Host-side plumbing like `releases`: it fetches only from `registry.npmjs.org`, through `releases._httpx_fetch`. Pure stdlib; no dspy.
+"""
+
+from __future__ import annotations
+
+import base64
+import io
+import json
+import logging
+import posixpath
+import re
+import tarfile
+from datetime import UTC, datetime
+from urllib.parse import quote, urlsplit
+
+from .releases import DownloadError, Fetch, ReleaseFile, ReleaseWatcher
+
+log = logging.getLogger(__name__)
+
+_PACKUMENT_URL = "https://registry.npmjs.org/{name}"
+_ACCEPT = "application/vnd.npm.install-v1+json"      # the abbreviated document npm itself installs from
+_MAX_PACKUMENT_BYTES = 64 << 20
+_MAX_ATTESTATION_BYTES = 4 << 20
+_MAX_MANIFEST_BYTES = 16 << 20     # one package.json
+_SLSA = "https://slsa.dev/provenance/v1"
+_NAME_RE = re.compile(r"(?:@[A-Za-z0-9][A-Za-z0-9._~-]{0,212}/)?[A-Za-z0-9][A-Za-z0-9._~-]{0,212}")
+_SEMVER_RE = re.compile(r"v?([0-9]{1,20})\.([0-9]{1,20})\.([0-9]{1,20})(?:-([0-9A-Za-z.-]{1,200}))?(?:\+[0-9A-Za-z.-]{1,200})?")
+_GITHUB_RE = re.compile(r"https://github\.com/([A-Za-z0-9][A-Za-z0-9-]{0,38}/[A-Za-z0-9._-]{1,100}?)(?:\.git)?/?")
+# the scripts npm runs when a package is installed from the registry (`prepare` runs only for a git or local install)
+_INSTALL_SCRIPTS = ("preinstall", "install", "postinstall")
+_YOUNG_DAYS = 30                   # a dependency created this recently is not yet a known package
+_MAX_DEPENDENCY_LOOKUPS = 50
+_MAX_FULL_DOCUMENT_BYTES = 64 << 20  # the full document gives the creation and publish times
+_DEPENDENCY_KEYS = ("dependencies", "optionalDependencies", "peerDependencies")      # npm 7 and later installs peers
+# a semver range or a dist-tag never holds `:` or `/`; a git, URL, path, `github:` or `npm:` alias spec does
+_OFF_REGISTRY_RE = re.compile(r"[:/\\]")
+
+
+class _Semver(tuple):
+    """A semver version's order: numbers, then a release after its pre-releases, whose identifiers order numbers first."""
+
+    @property
+    def is_prerelease(self) -> bool:
+        return not self[3]
+
+
+def semver_key(version: str) -> _Semver | None:
+    m = _SEMVER_RE.fullmatch(version)
+    if not m:
+        return None
+    pre = m.group(4)
+    ids = tuple((0, int(x), "") if x.isdigit() else (1, 0, x) for x in pre.split(".")) if pre else ()
+    return _Semver((int(m.group(1)), int(m.group(2)), int(m.group(3)), pre is None, ids))
+
+
+def parse_npm_packages(texts: list[str]) -> list[str]:
+    """npm package names (`left-pad`, `@scope/name`) from arguments and files: whitespace, comma or newline separated, `#` comments allowed."""
+    out: list[str] = []
+    for text in texts:
+        for line in text.splitlines() or [text]:
+            for word in line.split("#", 1)[0].replace(",", " ").split():
+                if len(word) > 214 or not _NAME_RE.fullmatch(word):
+                    raise ValueError(f"not an npm package name: {word!r}")
+                if word not in out:
+                    out.append(word)
+    return out
+
+
+def parse_packument(body: bytes) -> list[ReleaseFile]:
+    """Each version of an abbreviated packument as its one tarball, with the registry's sha512 and whether the registry holds its provenance. A tarball served from another host is left out: only the registry's are read."""
+    data = json.loads(body.decode("utf-8"))
+    out: list[ReleaseFile] = []
+    for version, meta in (data.get("versions") or {}).items():
+        dist = meta.get("dist") if isinstance(meta, dict) else None
+        if not isinstance(dist, dict) or semver_key(str(version)) is None:
+            continue
+        url = str(dist.get("tarball") or "")
+        if urlsplit(url).scheme != "https" or urlsplit(url).hostname != "registry.npmjs.org":
+            continue
+        integrity = str(dist.get("integrity") or "")
+        attestations = dist.get("attestations") if isinstance(dist.get("attestations"), dict) else {}
+        out.append(ReleaseFile(
+            filename=f"{posixpath.basename(urlsplit(url).path)}", url=url, version=str(version), kind="npm", size=0,
+            sha256="", upload_time="", yanked=False, attested=bool((attestations or {}).get("provenance")),
+            provenance=str(attestations.get("url") or "") if attestations else "",
+            sha512=integrity[len("sha512-"):] if integrity.startswith("sha512-") else ""))
+    return out
+
+
+def npm_publishers(fetch: Fetch, f: ReleaseFile, notes: list[str]) -> set[str]:
+    """The publisher a version's SLSA provenance names, as `releases.publisher_identity` would write it (`github:owner/name:.github/workflows/x.yml::`). The statement must name the tarball's own sha512, or it is a note."""
+    try:
+        status, _, body = fetch(f.provenance, {"Accept": "application/json"}, _MAX_ATTESTATION_BYTES)
+    except ValueError as exc:
+        notes.append(f"{f.filename}: its provenance was not read ({str(exc)[:120]})")
+        return set()
+    if status == 404:
+        return set()
+    if status != 200:
+        raise DownloadError(f"{f.filename}: its provenance answered HTTP {status}")
+    found: set[str] = set()
+    try:
+        for att in json.loads(body.decode("utf-8")).get("attestations") or []:
+            if att.get("predicateType") != _SLSA:
+                continue
+            statement = json.loads(base64.b64decode(att["bundle"]["dsseEnvelope"]["payload"]))
+            digests = {str((s.get("digest") or {}).get("sha512") or "") for s in statement.get("subject") or []}
+            if not f.sha512 or base64.b64decode(f.sha512).hex() not in digests:
+                notes.append(f"{f.filename}: its provenance names another tarball, or the registry gives no sha512 to bind it to")
+                continue
+            workflow = statement["predicate"]["buildDefinition"]["externalParameters"]["workflow"]
+            m = _GITHUB_RE.fullmatch(str(workflow.get("repository") or ""))
+            repo = m.group(1).lower() if m else str(workflow.get("repository") or "").lower()
+            kind = "github" if m else "other"
+            found.add(":".join(str(x).replace(":", "_") for x in (kind, repo, workflow.get("path") or "", "", "")))
+    except (ValueError, TypeError, KeyError, AttributeError):
+        notes.append(f"{f.filename}: its provenance could not be read")
+    return found
+
+
+def tarball_manifests(filename: str, data: bytes) -> tuple[dict[str, dict], set[str], list[str]]:
+    """Every `package.json` of a tarball as npm reads it, by the directory of its package (`""` the root, `node_modules/helper` a bundled dependency, whose install scripts npm runs too), with the package directories that hold a `binding.gyp`, and notes for what could not be read. As npm extracts: the first path component dropped (a member at the root is not installed), the last entry of a path standing, the bytes decoded as UTF-8 with a BOM dropped and invalid bytes replaced. Read from the tarball itself, never through the screen's text budget, and an unreadable manifest is a note (`release-not-fully-read`), never a package without scripts."""
+    from .releases import _MAX_MEMBERS, _MAX_READ_BYTES, _Capped, _decompressed
+
+    raw: dict[str, bytes | None] = {}
+    gyp: set[str] = set()
+    notes: list[str] = []
+    try:
+        stream = io.BufferedReader(_Capped(_decompressed(data), _MAX_READ_BYTES))
+        with tarfile.open(fileobj=stream, mode="r|") as tf:
+            for count, info in enumerate(tf, 1):
+                if count > _MAX_MEMBERS:
+                    notes.append(f"{filename}: more than {_MAX_MEMBERS} members, its manifests not read past them")
+                    break
+                parts = [p for p in info.name.split("/") if p not in ("", ".")]
+                if len(parts) < 2 or ".." in parts:          # npm refuses a path that climbs out
+                    continue
+                path = "/".join(parts[1:])
+                folder, base = posixpath.dirname(path), posixpath.basename(path).lower()
+                if not _package_dir(folder):
+                    continue
+                if base == "binding.gyp":                    # npm on a case-insensitive file system finds `Binding.gyp` too
+                    gyp.add(folder)
+                if base != "package.json":
+                    continue
+                if not info.isfile() or info.size > _MAX_MANIFEST_BYTES:
+                    raw[path] = None
+                    continue
+                fh = tf.extractfile(info)
+                raw[path] = fh.read() if fh is not None else None
+    except Exception as exc:  # noqa: BLE001 (fail CLOSED: a tarball whose manifests cannot be read is evidence)
+        notes.append(f"{filename}: its manifests were not read to the end ({type(exc).__name__})")
+    folded: dict[str, list[str]] = {}
+    for path in raw:
+        folded.setdefault(path.lower(), []).append(path)
+    for paths in folded.values():
+        if len(paths) > 1:      # which one npm reads depends on the file system: each is read, and the pair alerts
+            notes.append(f"{filename}: manifests whose paths differ only in case ({', '.join(sorted(paths))})"[:240])
+    manifests: dict[str, dict] = {}
+    for path, body in raw.items():
+        folder, where = posixpath.dirname(path), path
+        key = folder if posixpath.basename(path) == "package.json" else f"{folder} [{posixpath.basename(path)}]".strip()
+        if body is None:
+            notes.append(f"{filename}: {where} is not a regular file of at most {_MAX_MANIFEST_BYTES >> 20} MiB")
+            continue
+        try:
+            manifest = json.loads(body.decode("utf-8-sig", "replace"))
+        except (ValueError, RecursionError):
+            notes.append(f"{filename}: {where} could not be read as JSON")
+            continue
+        if isinstance(manifest, dict):
+            manifests[key] = manifest
+        else:
+            notes.append(f"{filename}: {where} is not a JSON object")
+    if "" not in manifests and not notes:
+        notes.append(f"{filename}: no package.json")
+    return manifests, gyp, notes
+
+
+def _package_dir(folder: str) -> bool:
+    """The root, or a dependency's directory under `node_modules`, at any depth (`node_modules/@s/x/node_modules/y`)."""
+    parts = folder.split("/") if folder else []
+    i = 0
+    while i < len(parts):
+        if parts[i] != "node_modules" or i + 1 >= len(parts):
+            return False
+        i += 3 if parts[i + 1].startswith("@") else 2
+        if i > len(parts):
+            return False
+    return True
+
+
+def _install_scripts(manifests: dict[str, dict], gyp: set[str]) -> dict[str, str]:
+    """The scripts an install from the registry runs, by `<package dir> <script>`: `preinstall`, `install`, `postinstall`, and the `node-gyp rebuild` npm runs for a `binding.gyp` when neither install script is given."""
+    found: dict[str, str] = {}
+    for folder, manifest in manifests.items():
+        scripts = manifest.get("scripts")
+        mine = {k: str(v) for k, v in (scripts.items() if isinstance(scripts, dict) else []) if k in _INSTALL_SCRIPTS}
+        if folder.split(" [", 1)[0] in gyp and not {"install", "preinstall"} & set(mine):
+            mine["install"] = "node-gyp rebuild (implied by binding.gyp)"
+        found.update({f"{folder} {k}".strip(): v for k, v in mine.items()})
+    return found
+
+
+def _dependencies(manifest: dict) -> set[str]:
+    return {str(name) for key in _DEPENDENCY_KEYS if isinstance(manifest.get(key), dict) for name in manifest[key]}
+
+
+def _registry_dependencies(manifests: dict[str, dict]) -> set[str]:
+    """Each dependency npm takes from the registry, by `<package dir> <name>`, in every manifest (a bundled package's missing dependencies are installed too)."""
+    return {f"{folder} {name}".strip() for folder, manifest in manifests.items() for key in _DEPENDENCY_KEYS
+            if isinstance(manifest.get(key), dict) for name, spec in manifest[key].items()
+            if not (_OFF_REGISTRY_RE.search(str(spec)) or str(spec).strip().startswith((".", "~/")))}
+
+
+def published_at(fetch: Fetch, package: str, version: str) -> datetime | None:
+    """When the registry says `version` of `package` was published (`time[version]` of its full document), so a dependency's age is measured as of the version, not the clock; None when that cannot be read."""
+    try:
+        status, _, body = fetch(_PACKUMENT_URL.format(name=quote(package, safe="@")), {"Accept": "application/json"},
+                                _MAX_FULL_DOCUMENT_BYTES)
+        return datetime.fromisoformat(str(json.loads(body.decode("utf-8"))["time"][version])) if status == 200 else None
+    except (ValueError, KeyError, TypeError, AttributeError):
+        return None
+
+
+def dependency_facts(fetch: Fetch, names: list[str], notes: list[str], now: datetime | None = None) -> dict:
+    """What the registry says of each dependency a version newly takes from it: one whose versions run a script at install (`hasInstallScript`), and one created within `_YOUNG_DAYS` before `now` (the version's publish time) or not on the registry at all. A compromised version need not carry the payload itself: it can depend on a package published hours before, whose install script runs it. A full document past its bound is a note, never an established package: an author pads a package's versions as easily as they publish them."""
+    now = now or datetime.now(UTC)
+    runs, young = [], []
+    if len(names) > _MAX_DEPENDENCY_LOOKUPS:
+        notes.append(f"{len(names)} new dependencies, the first {_MAX_DEPENDENCY_LOOKUPS} looked up")
+    for name in names[:_MAX_DEPENDENCY_LOOKUPS]:
+        if not _NAME_RE.fullmatch(name):
+            continue
+        url = _PACKUMENT_URL.format(name=quote(name, safe="@"))
+        status, _, body = fetch(url, {"Accept": _ACCEPT}, _MAX_PACKUMENT_BYTES)
+        if status == 404:
+            young.append(f"{name} (not on the registry)")
+            continue
+        if status != 200:
+            raise DownloadError(f"the registry's document of {name} answered HTTP {status}")
+        try:
+            versions = json.loads(body.decode("utf-8")).get("versions") or {}
+            scripted = sorted((v for v, meta in versions.items() if isinstance(meta, dict) and meta.get("hasInstallScript")),
+                              key=lambda v: (semver_key(v) is not None, semver_key(v) or (), v))
+        except (ValueError, AttributeError):
+            notes.append(f"the registry's document of {name} could not be read")
+            continue
+        if scripted:
+            runs.append(f"{name} ({len(scripted)} of {len(versions)} versions, {scripted[-1]} the last)"[:200])
+        try:
+            status, _, body = fetch(url, {"Accept": "application/json"}, _MAX_FULL_DOCUMENT_BYTES)
+        except ValueError:
+            notes.append(f"the registry's full document of {name} is past {_MAX_FULL_DOCUMENT_BYTES >> 20} MiB, so its age was not read")
+            continue
+        if status != 200:
+            raise DownloadError(f"the registry's full document of {name} answered HTTP {status}")
+        try:
+            created = datetime.fromisoformat(str(json.loads(body.decode("utf-8"))["time"]["created"]))
+        except (ValueError, KeyError, TypeError, AttributeError):
+            notes.append(f"the registry does not say when {name} was created")
+            continue
+        if (now - created).days < _YOUNG_DAYS:
+            young.append(f"{name} (created {created.date().isoformat()})")
+    return {"dependencies_run_at_install": runs, "dependencies_young": young}
+
+
+def _sources(manifests: dict[str, dict]) -> dict[str, str]:
+    """Each dependency npm installs from somewhere other than the registry, by `<package dir> <name>`: a git repository, a URL, a path, or an `npm:` alias to another package. npm fetches that code at install and runs its install scripts (a git dependency's `prepare` too), so a version needs no script of its own to run code."""
+    found: dict[str, str] = {}
+    for folder, manifest in manifests.items():
+        for key in _DEPENDENCY_KEYS:
+            deps = manifest.get(key)
+            for name, spec in (deps.items() if isinstance(deps, dict) else []):
+                spec = str(spec).strip()
+                if _OFF_REGISTRY_RE.search(spec) or spec.startswith((".", "~/")):
+                    found[f"{folder} {name}".strip()] = spec
+    return found
+
+
+def npm_facts(new: list[tuple[str, bytes]], old: list[tuple[str, bytes]], notes: list[str],
+              fetch: Fetch | None = None, package: str = "", version: str = "") -> dict:
+    """What a version runs at install and what it depends on, against the previous version, from the tarballs' manifests (`tarball_manifests`): an install script it adds (`npm-install-script-added`, high) or changes (`npm-install-script-changed`, medium), its own or a bundled dependency's, a dependency it now installs from outside the registry or from another such source (`npm-dependency-source`, high), and a new dependency (low). What of the new version's manifests could not be read is a note; the previous version's unread manifests count as none, so a script there is added. A new dependency from the registry is looked up there (`dependency_facts`)."""
+    if not new:
+        return {}
+    manifests, gyp = {}, set()
+    for filename, body in new:
+        found, dirs, unread = tarball_manifests(filename, body)
+        manifests.update(found)
+        gyp |= dirs
+        notes.extend(unread)
+    before_manifests, before_gyp = {}, set()
+    for filename, body in old:
+        found, dirs, _ = tarball_manifests(filename, body)
+        before_manifests.update(found)
+        before_gyp |= dirs
+    now, before = _install_scripts(manifests, gyp), _install_scripts(before_manifests, before_gyp)
+    sources, before_sources = _sources(manifests), _sources(before_manifests)
+    # a dependency newly taken from the registry, in any manifest: a new name, or one that came from elsewhere before
+    added = sorted({key.rsplit(" ", 1)[-1] for key in _registry_dependencies(manifests) - _registry_dependencies(before_manifests)})
+    looked = {}
+    if old and added and fetch is not None:
+        looked = dependency_facts(fetch, added, notes, published_at(fetch, package, version) if package else None)
+    return {
+        **looked,
+        "dependency_sources": [f"{k}: {v}"[:200] for k, v in sorted(sources.items()) if before_sources.get(k) != v]
+                              if old else [],
+        "install_scripts_added": [f"{k}: {v}"[:200] for k, v in sorted(now.items()) if k not in before] if old else [],
+        "install_scripts_changed": [f"{k}: {before[k]}"[:100] + f" -> {v}"[:100] for k, v in sorted(now.items())
+                                    if k in before and before[k] != v],
+        "new_requires": sorted(_dependencies(manifests.get("", {})) - _dependencies(before_manifests.get("", {})))[:50]
+                        if old else [],
+    }
+
+
+class NpmWatcher(ReleaseWatcher):
+    """`ReleaseWatcher` over the npm registry: the same per-file state, retries and alerts, in `npm.db`, with a version's tarball as its one file."""
+
+    registry = "npm"
+    state_file = "npm.db"
+
+    def _order(self, version: str):
+        return semver_key(version)
+
+    def _index(self, package: str, etag: str | None) -> tuple[list[ReleaseFile] | None, str | None]:
+        headers = {"Accept": _ACCEPT, **({"If-None-Match": etag} if etag else {})}
+        status, resp_headers, body = self.fetch(_PACKUMENT_URL.format(name=quote(package, safe="@")), headers,
+                                                _MAX_PACKUMENT_BYTES)
+        if status == 304:
+            return None, etag
+        if status != 200:
+            raise RuntimeError(f"the registry's document of {package} answered HTTP {status}")
+        return parse_packument(body), {k.lower(): v for k, v in resp_headers.items()}.get("etag")