@thomasminh1995/depverdict 0.6.0-alpha.1

package/docs/MVP-02-Research-Planning.md

@@ -0,0 +1,196 @@
+# MVP-02 Research Planning
+
+MVP-02-02 creates the deterministic bridge between an MVP-01 Project Manifest and later knowledge research. It reads a Project Manifest once, validates it, fingerprints its exact bytes, and produces an internal `ResearchPlan`.
+
+The Research Plan is internal. The [Knowledge Manifest](MVP-02-Knowledge-Manifest.md) is the portable public artifact produced only after later MVP-02 collection work completes.
+
+## Boundary
+
+```text
+Project Manifest bytes
+        │
+        ▼
+Parse + validate + exact-byte SHA-256
+        │
+        ▼
+Extract dependency occurrences
+        │
+        ▼
+Resolve public npm/PyPI identities
+        │
+        ▼
+Group + canonicalize + validate
+        │
+        ▼
+Internal Research Plan
+```
+
+This stage does not scan the repository. It does not read `package.json`, `requirements.txt`, lockfiles, or source files. It performs no network request, registry lookup, cache operation, source resolution, version comparison, or Knowledge Manifest generation.
+
+## Input and lineage
+
+`loadProjectManifestInput` accepts either:
+
+```js
+await loadProjectManifestInput('/path/to/project-manifest.json');
+
+await loadProjectManifestInput({
+  bytes: exactManifestBytes,
+  artifact: '.upgradelens/project-manifest.json'
+});
+```
+
+The file form reads the manifest once. The byte form is intended for controlled tests and internal callers. In both cases UpgradeLens hashes the exact byte sequence with SHA-256 before parsing it. It never hashes a reserialized JavaScript object.
+
+The resulting lineage is compatible with the future Knowledge Manifest input contract:
+
+```json
+{
+  "projectManifest": {
+    "schemaVersion": "2.0.0",
+    "artifact": ".upgradelens/project-manifest.json",
+    "artifactDigest": "sha256:<64 lowercase hexadecimal characters>",
+    "repository": { "name": "example-project", "root": "." }
+  }
+}
+```
+
+Artifact paths must be portable repository-relative paths. Absolute paths, drive-qualified paths, backslashes, and parent traversal are rejected. Invalid JSON, a non-`2.0.0` schema version, JSON Schema violations, and Project Manifest dependency-summary invariant violations are fatal input errors; they never produce a partial plan.
+
+## Project Manifest validation
+
+The loader uses `schemas/project-manifest.schema.json` with Ajv Draft 2020-12 strict mode. It also validates runtime relationships relevant to MVP-01 dependency inventories:
+
+- project, workspace, dependency-manifest paths are portable;
+- project and ecosystem summary counts match projects;
+- parsed dependency declaration, unique, and duplicate counts match inventory records;
+- Node `byType` counts match Node dependency types;
+- unsupported or failed dependency parsers have no inventory entries.
+
+The planner canonicalizes its own output, so a valid input with projects or dependencies in a different array order still produces the same package plan.
+
+## Internal Research Plan
+
+The internal plan is serializable for debugging and tests but is not written by default and has no public JSON Schema:
+
+```json
+{
+  "planVersion": "1",
+  "input": { "projectManifest": {} },
+  "summary": {
+    "inputProjectCount": 2,
+    "inputOccurrenceCount": 47,
+    "researchableOccurrenceCount": 46,
+    "uniqueResearchPackageCount": 45,
+    "invalidOccurrenceCount": 1,
+    "unsupportedOccurrenceCount": 0
+  },
+  "packages": [],
+  "invalidOccurrences": [],
+  "unsupported": [],
+  "warnings": []
+}
+```
+
+`planVersion` is independent from both Project Manifest and Knowledge Manifest schema versions. The plan has no timestamp, network/cache state, store detail, credentials, source content, or registry metadata.
+
+`inputOccurrenceCount` means every dependency declaration present in the valid Project Manifest inventories that planning examines: researchable and invalid Node/Python declarations plus declarations from unsupported ecosystems. Duplicate and multi-project declarations count separately. Parsed Node/Python inventories provide those records today; other ecosystems only contribute when a future Project Manifest contains an inventory for them.
+
+## Research identity
+
+MVP-02-02 supports only these public lookup identities:
+
+```text
+node    → npm:<normalized-name>
+python  → pypi:<normalized-name>
+```
+
+Normal npm names and scoped names such as `@vitejs/plugin-react` are valid. Python identities use the existing PEP 503-style normalization already used by MVP-01: lowercase and collapse runs of `.`, `_`, and `-` to `-`.
+
+Each research package is package-centric:
+
+```json
+{
+  "id": "npm:react",
+  "registry": "npm",
+  "ecosystem": "node",
+  "normalizedName": "react",
+  "observedDeclaredNames": ["react"],
+  "occurrences": []
+}
+```
+
+Researchable occurrences group only by `(registry, normalizedName)`. Grouping preserves every declaration, declared version/reference, dependency type, manifest path, project ID, and project path. It keeps duplicate declarations and declarations from separate projects; only the external lookup identity is deduplicated.
+
+Node ranges, tags, and other ordinary npm version declarations remain opaque. Python version declarations remain opaque as well: MVP-02-02 does not parse SemVer or PEP 440.
+
+## Eligibility and invalid references
+
+Node dependencies are researchable when their normalized name is a public npm name and their declaration is not a local path, workspace-only reference, file/link reference, npm alias, URL, Git reference, or similar unsupported reference.
+
+Python dependencies are researchable when they have an explicit named PyPI identity. A named direct reference such as `package-name @ https://…` is researchable using `package-name`. Unnamed URLs, unnamed Git references, and local editable paths without an explicit name are not guessed into a registry identity.
+
+Invalid Node/Python occurrences remain in `invalidOccurrences` with one stable reason:
+
+- `empty-name`
+- `invalid-npm-name`
+- `invalid-pypi-name`
+- `unnamed-direct-reference`
+- `local-path-reference`
+- `unsupported-reference`
+
+Each invalid occurrence produces an internal `INVALID_PACKAGE_REFERENCE` warning. URL user information, query strings, fragments, and local paths are sanitized before the Research Plan is serialized. UpgradeLens never guesses a package name from a URL, Git repository, path, or name similarity.
+
+## Unsupported ecosystems
+
+Java, .NET, Go, Rust, Ruby, PHP, AL, and other non-Node/non-Python dependencies are outside MVP-02-02. Their declarations are aggregated by ecosystem rather than copied into package records:
+
+```json
+{
+  "ecosystem": "java",
+  "projectIds": ["java:backend"],
+  "occurrenceCount": 12
+}
+```
+
+An ecosystem with one or more such declarations receives one internal `UNSUPPORTED_RESEARCH_ECOSYSTEM` warning. Valid dependencies from unsupported ecosystems are not reported as `INVALID_PACKAGE_REFERENCE`. No new public Knowledge Manifest warning code is introduced by this internal planning detail.
+
+## Determinism and invariants
+
+The plan uses code-unit lexical order:
+
+1. packages by ID;
+2. observed names lexically;
+3. occurrences by project ID, manifest, dependency type, declared name, then declared version (`null` sorts as an empty string);
+4. invalid occurrences by the same key and then reason;
+5. unsupported records by ecosystem and their project IDs lexically;
+6. warnings by scope, code, and message.
+
+`createResearchPlan` calls `validateResearchPlan` before returning. The validator fails fast when an internal implementation bug violates any of these relationships:
+
+- unique package count equals package records;
+- package occurrence total equals researchable occurrence count;
+- invalid and unsupported counts equal their records;
+- researchable + invalid + unsupported equals input occurrences;
+- package IDs match registry and normalized name;
+- package occurrences match their package identity;
+- warning references correspond to invalid or unsupported records;
+- IDs are unique, references are portable, output is sorted, and serialized values contain no URL credentials.
+
+## Public API
+
+The package exports only the planning entry points required by later MVP-02 implementation:
+
+```js
+import {
+  loadProjectManifestInput,
+  createResearchPlan,
+  validateResearchPlan
+} from 'upgradelens';
+```
+
+Call `loadProjectManifestInput` first, pass its result to `createResearchPlan`, and use `validateResearchPlan` to fail fast when inspecting or constructing an internal plan.
+
+## Deferred work
+
+This task deliberately does not implement npm/PyPI requests, GitHub API use, source resolution, Knowledge Store persistence, cache behavior, Knowledge Manifest creation, latest-version lookup, release metadata, version analysis, AI/LLM behavior, MCP, agent runtime, or a new CLI command.

package/docs/MVP-02-Source-Provenance.md

@@ -0,0 +1,81 @@
+# MVP-02 Source Provenance Resolution
+
+MVP-02-06 converts URL metadata already returned by the npm and PyPI registry adapters into a small, deterministic internal source graph. It does not fetch, crawl, or verify source content. It only structures and attributes source metadata already collected from registries.
+
+```text
+npm / PyPI registry result
+        │
+        ▼
+Extract source candidates
+        │
+        ▼
+Canonicalize and classify public URLs
+        │
+        ▼
+Merge compatible candidates per package
+        │
+        ▼
+Attach provenance, trust, and conflicts
+        │
+        ▼
+Internal source graph
+```
+
+For example:
+
+```text
+npm Registry
+      │
+      ├── documentation candidate ──► npm:react:documentation:<sha256>
+      └── repository candidate ─────► npm:react:repository:<sha256>
+```
+
+The source graph is internal composition data. It is not a Knowledge Manifest, a cache format, or a graph database. Later orchestration can assemble compatible source records into the versioned Knowledge Manifest without exposing adapter payloads, source-candidate field paths, or cache details.
+
+## URL evidence and provenance
+
+A URL alone is not proof that a project controls the destination. Registry metadata provides a publisher assertion about a relationship; it does not establish domain or repository ownership. Each non-registry source records its source kind, supported roles, authority, trust, status, one deterministic upstream `discoveredFrom` source ID, and all upstream source IDs in `trustEvidenceSourceIds`.
+
+The public Knowledge Manifest contract permits one `discoveredFrom` source ID rather than a field-path object. The resolver chooses it by canonical source ID and field-path ordering. Field paths are retained only while resolving candidates and do not become a public artifact. This preserves portability while still retaining all distinct upstream source IDs as trust evidence.
+
+## Roles, kinds, and status
+
+| Candidate role | Knowledge Manifest kind | `supports` value |
+| --- | --- | --- |
+| homepage | `officialDocumentation` | `homepage` |
+| documentation | `officialDocumentation` | `documentation` |
+| repository | `sourceRepository` | `repository` |
+| issues | `sourceRepository` | `issues` |
+| changelog | `officialDocumentation` | `changelog` |
+| release notes | `officialDocumentation` | `releaseNotes` |
+| releases | `releaseFeed` | `releases` |
+
+Registry sources remain adapter-owned, such as `npm:react:registry`, with `registryAuthoritative` authority and `publisher` trust. Registry facts are authoritative for their registry, but publisher-entered URL fields default to `publisherProvided`, `publisher`, and `unverified`. No URL is labelled `official` or `officialProject` from package name, organization name, or domain similarity.
+
+`verified` is possible only when two distinct registry source IDs assert the same canonical relationship for the same package ID. Duplicate fields in one registry record are not independent corroboration. Package IDs are intentionally not merged across npm and PyPI merely because their URLs match; cross-ecosystem project equivalence is deferred.
+
+## Canonical public URLs and source IDs
+
+Only portable HTTPS URLs are emitted. Resolution rejects malformed values, HTTP, non-web schemes, embedded credentials, query strings, localhost, loopback, link-local and private IP literals. Fragments are removed. Hosts are normalized by the URL parser, root and path trailing slashes are normalized consistently, `git+https:` repository values become HTTPS, and a GitHub repository `.git` suffix is removed. No DNS lookup, redirect, or request occurs. Meaningful paths remain distinct: a GitHub repository and its `/issues` page are separate records.
+
+Non-registry IDs are package-scoped and role-qualified:
+
+```text
+npm:react:documentation:<sha256(canonical-url)>
+npm:react:repository:<sha256(canonical-url)>
+pypi:fastapi:changelog:<sha256(canonical-url)>
+```
+
+They contain no local path, cache key, timestamp, credential, or random value. The same package, compatible role group, and canonical URL always produce the same ID. Homepage and documentation use one compatible documentation group when the URL is identical, preserving both sorted support roles. Repository and issue URLs remain separate even when hosted by the same service.
+
+## Deduplication and conflict behavior
+
+Candidates merge only when package ID, canonical URL, and compatible role group match. Different roles do not conflict automatically. If the same package asserts two distinct URLs for the same role, the resolver keeps both records, adds symmetric sorted `conflictsWith` references, and emits a non-retryable `SOURCE_CONFLICT` warning for each conflicting record. It never chooses one URL silently or attempts semantic conflict resolution.
+
+All packages, source IDs, sources, supports, trust evidence, conflict references, and warnings use code-unit lexical ordering. The graph therefore does not depend on adapter completion order.
+
+## Privacy, safety, and limitations
+
+The graph contains normalized public URLs and no raw registry payload, response headers, source content, credentials, query tokens, local paths, or cache implementation details. Unsafe candidates are omitted without a public warning because no fetch occurred to establish that a source is missing.
+
+MVP-02-06 intentionally does not fetch websites, crawl documentation, call GitHub APIs or Releases, follow redirects, verify ownership, discover community resources, calculate numeric trust scores, assemble a Knowledge Manifest, add a CLI command, compare versions, analyze breaking changes, or perform AI/LLM reasoning.

package/docs/MVP-02-npm-Registry-Adapter.md

@@ -0,0 +1,99 @@
+# MVP-02 npm-Compatible Registry Adapter
+
+MVP-02-04 adds the internal adapter that collects facts for Node package identities in the npm-compatible registry ecosystem. It accepts a research package from the deterministic Research Plan and returns internal package, registry-source, cache-outcome, and warning records. It does not assemble or write a Knowledge Manifest.
+
+> The `npm:` package identity identifies the npm-compatible registry ecosystem. It does not mean that the source repository uses the npm CLI.
+
+## Package managers and registry identity
+
+Projects using npm, Yarn, pnpm, or Bun commonly resolve the same public Node package identity through an npm-compatible registry. The adapter therefore works only from:
+
+```text
+registry: npm
+packageId: npm:<normalized-package-name>
+```
+
+It does not invoke a package-manager CLI, inspect lockfiles, or read `.npmrc`, `.yarnrc.yml`, or `bunfig.toml`. Package-manager origin is intentionally absent from both the registry request and the cache identity. Equivalent `npm:react` Research Plan records produce the same request, cache entry, and normalized facts regardless of whether their source project used npm, Yarn, pnpm, or Bun.
+
+## Public Registry scope and requests
+
+The default approved Registry base is `https://registry.npmjs.org`. Internal callers may supply another explicit HTTPS base for tests or a future approved policy. HTTP, credentialed URLs, query-bearing bases, and repository-derived registry selection are rejected.
+
+The adapter requests the full packument representation with one deterministic `GET`:
+
+```text
+<registry-base>/<encodeURIComponent(normalized-package-name)>
+```
+
+Examples:
+
+```text
+https://registry.npmjs.org/react
+https://registry.npmjs.org/%40vitejs%2Fplugin-react
+```
+
+The adapter uses `Accept: application/json`, a stable UpgradeLens user agent, `credentials: omit`, and disabled redirects. It sends no authorization or cookie headers. npm search is never used as an identity fallback.
+
+## Cache integration
+
+The adapter uses the private [Lightweight Knowledge Store](MVP-02-Knowledge-Store.md) with this identity:
+
+```json
+{
+  "adapter": "npm",
+  "resourceKind": "registry-package",
+  "packageId": "npm:react",
+  "resourceVariant": "full-packument",
+  "adapterVersion": "1"
+}
+```
+
+The default adapter TTL is 24 hours, but callers may provide a policy TTL. The store itself remains registry-agnostic.
+
+| Cache state | Adapter behavior |
+| --- | --- |
+| Fresh | Normalize the cached packument; no request is made. |
+| Missing | Fetch, validate, atomically cache, then normalize. |
+| Expired | Fetch and replace on success. Fetch failure returns unavailable facts; stale data is not silently used. |
+| Corrupted | Treat as unusable, fetch a replacement, and overwrite only after validation succeeds. |
+
+The internal result records `hit`, `miss`, `revalidated`, or a corruption-replacement outcome without exposing filenames, cache keys, or store paths.
+
+## Packument facts and normalization
+
+The adapter validates that the response has the requested exact package name, `dist-tags`, `versions`, and `time` objects before caching it. A scoped name must match exactly; a response for another package is rejected. Optional malformed publisher metadata is omitted rather than causing a crash.
+
+It normalizes these facts when safely present:
+
+- description and license;
+- HTTPS homepage, repository, and issue URLs;
+- package-level deprecation message/status when explicitly reported;
+- `dist-tags.latest` as a registry fact;
+- a minimal release index with opaque version, valid publication time, npm release URL, and explicit deprecation state.
+
+Repository strings and `{ "url": "..." }` objects are supported. `git+https:` is normalized to HTTPS, GitHub `.git` suffixes are removed, and credentialed repository URLs are dropped. URL queries and fragments are not retained.
+
+`latest` is taken only from `dist-tags.latest`; the adapter never chooses a lexical or semantic maximum. If that tag is absent or references a version missing from `versions`, no alternative version is selected and the result is partial. Release versions are sorted by code-unit lexical order. The adapter does not parse SemVer, infer prereleases, or treat a version as yanked without an explicit registry fact.
+
+Each successful result creates one internal registry source record such as `npm:react:registry`, with registry authority, publisher trust, package/API URLs, source roles, content digest, retrieval timestamp, and `fresh` snapshot state. This record is shaped for later Knowledge Manifest assembly but is not itself a manifest.
+
+## Response limits and error mapping
+
+Full npm packuments contain complete metadata for every published version and can be several megabytes. The adapter therefore owns a default bounded limit of **16 MiB** (`16 * 1024 * 1024`). Callers may set `maxResponseBytes` explicitly for tests or a future execution policy; it must be a positive safe integer and may not exceed the documented 64 MiB safety ceiling. The setting is execution configuration, not Knowledge Manifest policy.
+
+Response bodies are read through a bounded streaming reader; the adapter never calls `response.json()` directly. It requires a JSON media type and rejects a body larger than the configured maximum without caching it. An oversized body retains the internal `NPM_RESPONSE_TOO_LARGE` code and maps to the existing public `REGISTRY_RESPONSE_INVALID` warning.
+
+Every response body is fully consumed or explicitly cancelled. Unused non-200 bodies, wrong-media bodies, and oversized streams are cancelled; oversized readers are cancelled and release their lock. The request timeout covers the complete fetch/read lifecycle and is cleared in `finally` on every outcome. Online CLI research injects a scoped, CLI-owned fetch and closes its Agent after completion; direct callers retain ownership of injected fetch resources.
+
+| Condition | Package status | Warning |
+| --- | --- | --- |
+| Registry `404` | `notFound` | `PACKAGE_NOT_FOUND` |
+| Registry `429` | `unavailable` | `REGISTRY_RATE_LIMITED` |
+| `5xx`, timeout, or transport failure | `unavailable` | `REGISTRY_UNAVAILABLE` |
+| Invalid JSON/media/size/packument shape/name mismatch | `unavailable` | `REGISTRY_RESPONSE_INVALID` |
+
+Negative responses, rate-limit responses, invalid payloads, and transport failures are not cached.
+
+## Intentional limitations
+
+This adapter does not implement private registries, authentication, package-manager configuration, package-manager CLI execution, lockfile parsing, Yarn/pnpm/Bun-specific adapters, PyPI, GitHub Releases, documentation crawling, conditional HTTP requests, retries, Knowledge Manifest orchestration, a research CLI command, version comparison, breaking-change analysis, or AI behavior.