narvy-cli 1.1.3__tar.gz → 1.1.5__tar.gz

{narvy_cli-1.1.3 → narvy_cli-1.1.5}/CHANGELOG.md

@@ -5,6 +5,144 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.1.5]
+
+### Changed
+
+- The dependency check (SCA) now queries Narvy's EU-hosted vulnerability
+  database, a mirror of OSV.dev, instead of api.osv.dev and api.github.com.
+  It sends package names, ecosystems and versions only, never code, file
+  paths or findings, and says so when it starts. `OSV_API_URL` points it
+  at another OSV-compatible server; it accepts a base URL or, as before,
+  the full `/query` URL. The batch URL now follows `OSV_API_URL` (it used
+  to stay on api.osv.dev unless `OSV_BATCH_API_URL` was also set).
+- iOS: the direct GitHub Advisory lookup is off by default. All 63 reviewed
+  GitHub `swift` advisories are in OSV as `SwiftURL`, so the database above
+  carries them. `NARVY_GITHUB_ADVISORIES=1` turns the direct lookup back on,
+  and it now uses `GITHUB_TOKEN` when set (the rate-limit message already
+  told users to set it, but it was not read).
+- Batch pre-filter requests carry up to 1000 dependencies and stay under
+  1 MB. A batch the server refuses as too large (HTTP 413) is split in
+  halves and retried, down to a single dependency; one that is still
+  refused is looked up on its own and counted unresolved if that fails.
+- A scan now checks up to 20,000 unique dependencies (was 400, which left
+  projects such as a Django app with a JavaScript front end reported as
+  partial). Past that cap the check is still reported partial, with the
+  number not queried. Requests to Narvy's database are paced to stay under
+  its per-IP limits, so a very large project waits instead of being cut
+  off by a rate limit; small scans are not slowed.
+
+### Fixed
+
+- A dependency check that got no usable answer is reported INCOMPLETE,
+  never clean, in every case: server unreachable, 503 (database data older
+  than 48 hours, or unavailable), 429 after one retry that honours
+  `Retry-After` (at most 10 s), 400, 404 or an unreadable response. After
+  the first such failure the rest of the run stops querying and counts the
+  remaining dependencies as unresolved, with one log line for the outage
+  instead of one per dependency. The end-of-scan message names the server
+  and the reason.
+- iOS libraries with no known source repository URL (most CocoaPods pods
+  and embedded frameworks) were reported with 0 CVEs as if checked. They
+  are now looked up by bare name (a few advisories are published that way)
+  and counted as unverified in the SCA coverage note.
+- Ecosystems the database does not carry are no longer sent (the server
+  would reject the whole batch); those dependencies are counted as
+  unverified.
+- Results served from an expired local cache while the database was
+  unavailable are now flagged in the coverage note.
+- JSON output: `summary.sca_coverage` gains `dependencies_unverified`,
+  `dependencies_from_stale_cache` and `vulnerability_db` (server host,
+  database snapshot time and age when the server reports them, last error).
+- Android: an APK scan is no longer refused for lack of memory. 1.1.2 to
+  1.1.4 refused a 6,300-class app on a laptop with 1.1 GB free ("--max-mem
+  4g asks JADX for 4.0 GB of Java heap ... run it anyway with --force"),
+  and the `--max-mem 1g` it suggested was refused the same way. The jadx
+  heap is now sized for each app: the smallest of `--max-mem` (new default
+  `auto`), what fits in the memory free right now, and twice the app's
+  estimate. The JVM's own memory is counted as about 250 MB plus 5% of the
+  heap (measured on real apps), not a fixed 1 GB reserve plus 30%. When the
+  app's estimate does not fit, the scan runs with the largest heap that
+  does and says so before jadx starts. If jadx runs out of Java heap it is
+  retried once with a bigger heap when memory allows; when the operating
+  system refuses memory (commit limit, OOM killer) it is retried once with
+  a smaller heap and one thread; then the scan fails with the numbers and
+  what to do. `--force` now means: exactly `--max-mem`, no sizing, no retry.
+- A generous `--max-mem` no longer inflates memory use: `-Xmx4g` on a
+  6.5k-class app peaked at 1.56 GB of RAM against 0.99 GB with the 1 GB
+  heap now chosen, for the same decompiled output.
+- Linux and macOS: jadx no longer runs under an address-space limit
+  (RLIMIT_AS). With a small heap it stopped the JVM from starting its
+  threads, and jadx then exited 0 without writing any source, which would
+  have been scanned as an empty app. A jadx run that exits 0 without
+  writing sources, or that printed an OutOfMemoryError, is now a failure,
+  never a clean scan.
+- Linux: free memory honours a container or cgroup memory limit (Docker
+  `--memory`, CI runners, systemd `MemoryMax`); `/proc/meminfo` shows the
+  whole host there, so a 2 GB container used to size jadx for the host.
+- Windows: free memory is `ullAvailPhys` (Task Manager's "Available",
+  standby cache included), now also bounded by the commit charge left.
+- Split bundles (.apkm/.xapk/.apks): the heap estimate counts the feature
+  modules jadx decompiles together with the base APK.
+- The line printed after decompiling gives the heap used and how many
+  methods jadx could not fully decompile (kept as low-level code, so
+  findings inside them can be missed); a tight heap raises that number.
+- JSON and SARIF output say whether the structural (Semgrep) pass
+  completed: `summary.structural_coverage` and
+  `runs[0].properties.structuralCoverage` (`complete`, `incomplete` with
+  the reason, or `not_run`). A timed-out pass was only visible on the
+  console.
+- Messages no longer send users to `--upload` for something it cannot do:
+  it sends findings only, so it neither decompiles a large app nor finishes
+  a timed-out structural pass. The memory messages point to an upload from
+  the Narvy dashboard, the structural-pass note to `--upload-binary`.
+- The SCA result line no longer reads "checked N ..., found 0 known
+  CVE(s)" when the lookups failed; it says their CVE status is unknown.
+- A jadx failure message is printed as text: brackets in jadx output were
+  read as console markup.
+
+## [1.1.4]
+
+### Fixed
+
+- APK scans: the structural (Semgrep) pass could match none of the app's
+  own files and report "Deep analysis INCOMPLETE" (seen on Windows with
+  Python 3.10). Two ways to get there were reproduced: a scan folder
+  reached through a symlink (a Windows 8.3 short name such as `RUNNER~1`
+  or a junction gives the folder a second spelling in the same way), and a
+  `.git`/`.hg`/`.svn` folder in any parent of the scan folder, which moves
+  semgrep's project root above it so the `sources/...` path filter no
+  longer matched. The scan folder is now resolved to one long-name,
+  physical path for every tool and every relative path, the path filters
+  match at any depth, and if they still match nothing the app's own
+  package folders are scanned as explicit roots.
+- Secret findings that fired only on a variable NAME (key, secret,
+  password, token, credential) were CRITICAL whatever the value was.
+  The value is now checked: a known provider credential format (AWS,
+  Google, Stripe, GitHub, Slack, ...) or key material of a real key length
+  keeps the rule's severity; a plain non-secret (URL, path, dotted or
+  snake_case identifier, UPPER_CASE constant, label repeating the word
+  "password", format string, regex, sentence, placeholder) is dropped;
+  anything else is capped at MEDIUM with LOW confidence and says why.
+  Applies to Android (APK and source), iOS Objective-C and Swift source,
+  and iOS binary strings. Provider-format rules are unchanged. JSON output
+  carries `original_severity` and `value_evidence` for re-graded findings.
+
+### Changed
+
+- End-of-scan message: "Scanned ... with N local rules" counted only the
+  pattern rules (APK) or mixed in one entry per dependency advisory found
+  (source). It now gives the number of rules actually executed, by engine
+  (pattern, taint, structural, native-library, config), and separately how
+  many of them produced findings and how many dependency advisories
+  matched. A structural pass that did not complete is not counted. JSON
+  output: `summary.rules_executed`, `rules_executed_by_engine`,
+  `rules_matched`, `sca_advisories_matched`.
+- Console summary table: hits of the same rule in the same file are one
+  row with a hit count and the lines involved. `--no-group` restores one
+  row per hit. SARIF and JSON output still list every hit.
+- Platform CI harness prints `narvy doctor` output.
+
 ## [1.1.3]
 
 ### Fixed

{narvy_cli-1.1.3/narvy_cli.egg-info → narvy_cli-1.1.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: narvy-cli
-Version: 1.1.3
+Version: 1.1.5
 Summary: Free, local SAST scanner for Android, iOS, and polyglot web/backend source - plus passive web, host-audit, and cloud-posture scanning.
 Author: Narvy
 License: Apache-2.0
@@ -60,6 +60,8 @@ Your code, binaries and findings are never uploaded unless you explicitly
 pass `--upload`. The CLI does send anonymous usage statistics, described
 exactly in [Telemetry](#telemetry), which you can turn off.
 
+The dependency check sends package names and versions (never code, file paths or findings) to Narvy's EU-hosted vulnerability database. Set OSV_API_URL to use another OSV-compatible server.
+
 ## Install
 
 ```bash
@@ -185,14 +187,22 @@ Run `narvy doctor` before your first scan to check your Java/jadx/iOS
 tooling setup, and `narvy <command> --help` for the full flag list of
 any subcommand.
 
+The console summary table groups hits of the same rule in the same file
+into one row, with a hit count and the lines involved. `--no-group` shows
+one row per hit. SARIF and JSON output always list every hit.
+
 ### Very large Android apps
 
 Decompiling is memory-hungry, and file size is a poor predictor of how much
 you need. Before decompiling, the CLI reads the DEX headers to estimate the
-Java heap the app needs and checks it against `--max-mem` and your
-machine's free memory, so an app that won't fit fails in about a second
-instead of after several minutes of doomed work. The estimate is a
-heuristic, not a hard limit - `--force` runs the decompile regardless.
+Java heap the app needs and sizes the jadx heap from that estimate and the
+memory free right now: a small app on a busy 8 GB laptop gets a small heap
+instead of a refusal. `--max-mem` sets an upper limit (default `auto`). When
+the estimate does not fit in free memory the scan still runs with the
+largest heap that does, and says so. If jadx runs out of memory it is
+retried once (a bigger heap when memory allows, otherwise one worker
+thread), then the scan fails with the numbers. `--force` runs exactly
+`--max-mem`, without sizing or retry.
 
 ### Sending results to your Narvy dashboard
 
@@ -227,6 +237,29 @@ exhaust the machine. `NARVY_SEMGREP_JOBS` and `NARVY_SEMGREP_MAX_MEMORY_MB`
 listed in the scan notes. Outside a terminal (CI logs), a progress line is
 printed every 30 seconds; `NARVY_PROGRESS_INTERVAL` changes that (0 = off).
 
+## Dependency check (SCA)
+
+The dependency check sends package names and versions (never code, file paths or findings) to Narvy's EU-hosted vulnerability database. Set OSV_API_URL to use another OSV-compatible server.
+
+What is sent, per dependency: the package name, its ecosystem (npm, PyPI,
+Maven, ...) and the resolved version. Nothing else: no code, no file paths,
+no project or repository name, no findings. The database is a mirror of
+[OSV.dev](https://osv.dev) (data from OSV.dev and the sources it
+aggregates, including the GitHub Advisory Database), refreshed from the
+OSV.dev export and hosted in the EU by Narvy.
+
+`OSV_API_URL` takes a base URL (`/query` and `/querybatch` are appended,
+e.g. `OSV_API_URL=https://api.osv.dev/v1`) or, as in earlier releases, the
+full query URL ending in `/query`. The batch URL follows it unless
+`OSV_BATCH_API_URL` is set.
+
+The check is reported INCOMPLETE, never clean, when the database cannot give
+a usable answer: unreachable, rate-limited, or its data older than 48 hours.
+Dependencies it cannot look up (an ecosystem the database does not carry, an
+iOS library with no known source repository URL) are counted as unverified
+in the scan's coverage note. Results are cached for 7 days in
+`~/.narvy/osv_cache.db`.
+
 ## Telemetry
 
 The CLI sends one small anonymous event per command, so we can see which

{narvy_cli-1.1.3 → narvy_cli-1.1.5}/README.md

@@ -27,6 +27,8 @@ Your code, binaries and findings are never uploaded unless you explicitly
 pass `--upload`. The CLI does send anonymous usage statistics, described
 exactly in [Telemetry](#telemetry), which you can turn off.
 
+The dependency check sends package names and versions (never code, file paths or findings) to Narvy's EU-hosted vulnerability database. Set OSV_API_URL to use another OSV-compatible server.
+
 ## Install
 
 ```bash
@@ -152,14 +154,22 @@ Run `narvy doctor` before your first scan to check your Java/jadx/iOS
 tooling setup, and `narvy <command> --help` for the full flag list of
 any subcommand.
 
+The console summary table groups hits of the same rule in the same file
+into one row, with a hit count and the lines involved. `--no-group` shows
+one row per hit. SARIF and JSON output always list every hit.
+
 ### Very large Android apps
 
 Decompiling is memory-hungry, and file size is a poor predictor of how much
 you need. Before decompiling, the CLI reads the DEX headers to estimate the
-Java heap the app needs and checks it against `--max-mem` and your
-machine's free memory, so an app that won't fit fails in about a second
-instead of after several minutes of doomed work. The estimate is a
-heuristic, not a hard limit - `--force` runs the decompile regardless.
+Java heap the app needs and sizes the jadx heap from that estimate and the
+memory free right now: a small app on a busy 8 GB laptop gets a small heap
+instead of a refusal. `--max-mem` sets an upper limit (default `auto`). When
+the estimate does not fit in free memory the scan still runs with the
+largest heap that does, and says so. If jadx runs out of memory it is
+retried once (a bigger heap when memory allows, otherwise one worker
+thread), then the scan fails with the numbers. `--force` runs exactly
+`--max-mem`, without sizing or retry.
 
 ### Sending results to your Narvy dashboard
 
@@ -194,6 +204,29 @@ exhaust the machine. `NARVY_SEMGREP_JOBS` and `NARVY_SEMGREP_MAX_MEMORY_MB`
 listed in the scan notes. Outside a terminal (CI logs), a progress line is
 printed every 30 seconds; `NARVY_PROGRESS_INTERVAL` changes that (0 = off).
 
+## Dependency check (SCA)
+
+The dependency check sends package names and versions (never code, file paths or findings) to Narvy's EU-hosted vulnerability database. Set OSV_API_URL to use another OSV-compatible server.
+
+What is sent, per dependency: the package name, its ecosystem (npm, PyPI,
+Maven, ...) and the resolved version. Nothing else: no code, no file paths,
+no project or repository name, no findings. The database is a mirror of
+[OSV.dev](https://osv.dev) (data from OSV.dev and the sources it
+aggregates, including the GitHub Advisory Database), refreshed from the
+OSV.dev export and hosted in the EU by Narvy.
+
+`OSV_API_URL` takes a base URL (`/query` and `/querybatch` are appended,
+e.g. `OSV_API_URL=https://api.osv.dev/v1`) or, as in earlier releases, the
+full query URL ending in `/query`. The batch URL follows it unless
+`OSV_BATCH_API_URL` is set.
+
+The check is reported INCOMPLETE, never clean, when the database cannot give
+a usable answer: unreachable, rate-limited, or its data older than 48 hours.
+Dependencies it cannot look up (an ecosystem the database does not carry, an
+iOS library with no known source repository URL) are counted as unverified
+in the scan's coverage note. Results are cached for 7 days in
+`~/.narvy/osv_cache.db`.
+
 ## Telemetry
 
 The CLI sends one small anonymous event per command, so we can see which

{narvy_cli-1.1.3 → narvy_cli-1.1.5}/narvy/__init__.py

@@ -1,3 +1,3 @@
 """Narvy CLI - free, local SAST for Android, iOS and polyglot source."""
 
-__version__ = "1.1.3"
+__version__ = "1.1.5"

{narvy_cli-1.1.3 → narvy_cli-1.1.5}/narvy/android/source_analyzer.py

@@ -6,6 +6,7 @@ import re
 from typing import Any, Dict, List, Optional, Tuple
 
 from narvy.rule_engine import load_rules_from_dir, run_rules_on_file
+from narvy.pathnorm import canonical_path
 from narvy import semgrep_engine
 from narvy import crypto_taint_lite
 from narvy.scope_config import ScopeConfig
@@ -265,6 +266,7 @@ def analyze_source(source_dir: str, override_config: Optional[ScopeConfig] = Non
             **empty_stats,
         }
 
+    source_dir = canonical_path(source_dir)
     notes: List[str] = []
     all_findings: List[Dict[str, Any]] = []
 
@@ -343,6 +345,9 @@ def analyze_source(source_dir: str, override_config: Optional[ScopeConfig] = Non
         )
 
     rule_defs = list(rules) + semgrep_rule_defs
+    rules_run = {"pattern": len(rules), "taint": len(crypto_taint_lite.RULE_IDS)}
+    if semgrep_rule_defs and semgrep_engine.LAST_RUN.get("status") == "ok":
+        rules_run["structural"] = len(semgrep_rule_defs)
 
     all_findings = _drop_prefkey_echo_secrets(source_dir, all_findings)
     all_findings = _downrank_test_findings(all_findings)
@@ -352,6 +357,7 @@ def analyze_source(source_dir: str, override_config: Optional[ScopeConfig] = Non
         "error": None,
         "findings": all_findings,
         "rule_defs": rule_defs,
+        "rules_run": rules_run,
         "manifests_found": len(manifest_paths),
         "own_roots": sorted(own_roots),
         "code_files_scanned": code_files_scanned,

{narvy_cli-1.1.3 → narvy_cli-1.1.5}/narvy/android_rule_context.py

@@ -4,6 +4,8 @@ from __future__ import annotations
 import re
 from typing import Any, Dict, Optional
 
+from . import secret_value_grade
+
 # PendingIntent flags; jadx emits these as decimals (67108864 / 33554432).
 _FLAG_IMMUTABLE = 0x04000000
 _FLAG_MUTABLE = 0x02000000
@@ -176,11 +178,28 @@ def _google_api_key_gate(content: str, match: "re.Match",
     return out
 
 
+_UNQUOTED_VALUE_RE = re.compile(r"[:=][\s\"']*([^\s\"',;<>)]+)")
+
+
+def _name_only_secret_gate(content: str, match: "re.Match",
+                           finding: Dict[str, Any]) -> Optional[Dict[str, Any]]:
+    """Name-matched secret rules: grade the assigned value (see secret_value_grade)."""
+    line_end = content.find("\n", match.start())
+    text = content[match.start():line_end if line_end != -1 else len(content)]
+    value = secret_value_grade.literal_from_text(text)
+    if value is None:
+        m = _UNQUOTED_VALUE_RE.search(match.group(0))
+        value = m.group(1) if m else None
+    return secret_value_grade.apply(dict(finding), value)
+
+
 RULE_GATES = {
     "AND-CONF-009": _pending_intent_gate,
     "AND-CONF-004": _exported_component_gate,
     "AND-S-005": _google_api_key_gate,
 }
+for _rid in ("AND-S-001", "AND-S-002", "AND-S-010", "IOS-OBJC-S-001", "IOS-OBJC-S-002"):
+    RULE_GATES[_rid] = _name_only_secret_gate
 
 
 def apply_gate(rule_id: str, content: str, match: "re.Match",