freshcontext-mcp 0.3.19 → 0.3.20

package/docs/HA_PRI_V2_PRODUCTION_ENFORCEMENT_PLAN.md

@@ -0,0 +1,414 @@
+# Ha-Pri v2 Production Enforcement Plan
+
+Status: design only
+Phase: Pass 11-K
+Runtime impact: none
+
+Ha-Pri v2 production enforcement is a future rollout path. Current FreshContext releases only include the Core helper and deterministic golden vectors unless a later implementation pass explicitly wires Worker/D1 enforcement.
+
+This plan describes how Ha-Pri v2 could move from pure Core provenance helper to production Store/Worker verification without overclaiming current behavior.
+
+## 1. Current State
+
+Current FreshContext behavior:
+
+- Ha-Pri v1 is the current Worker/feed audit stamp.
+- Ha-Pri v1 is stored as `scrape_results.ha_pri_sig`.
+- Ha-Pri v1 is returned in Worker feed `intelligence_stamps`.
+- Ha-Pri v1 is a provenance stamp and audit reference, not hard row rejection.
+- Ha-Pri v2 exists as a pure Core helper in `src/core/provenance.ts`.
+- Ha-Pri v2 has deterministic golden vectors in `tests/fixtures/ha-pri-v2-golden-vectors.json`.
+- Ha-Pri v2 is not production-enforced on Worker/D1 reads.
+- Ha-Pri v2 does not currently reject rows.
+- Ha-Pri v2 does not currently provide private-key origin authentication.
+
+The current v2 helper provides deterministic canonicalization, SHA-256 hashing, signing payload construction, signature calculation, and verification status:
+
+```txt
+valid
+invalid
+unknown
+```
+
+That is real Core behavior. It is not yet Worker/D1 enforcement.
+
+## 2. Target Future State
+
+Future production enforcement should make stored context rows independently reviewable after write.
+
+Target behavior:
+
+- Worker write path stores Ha-Pri v2 provenance material for new rows.
+- Rows can later be verified as `valid`, `invalid`, or `unknown`.
+- Debug/internal read paths can report verification status.
+- Safe public read paths can expose limited verification status without leaking internals.
+- Invalid rows are not silently treated as trusted.
+- Old rows remain compatible through `unknown` and optional backfill behavior.
+- Strict rejection remains optional and staged, not the first rollout.
+
+The target is not "FreshContext proves truth." The target is "FreshContext can detect whether stored provenance material still matches the stored row material under the documented v2 contract."
+
+## 3. Proposed D1 / Storage Fields
+
+Essential fields:
+
+```txt
+ha_pri_sig_v2 TEXT
+ha_pri_canonical_content_sha256 TEXT
+ha_pri_semantic_fingerprint_sha256 TEXT
+ha_pri_signing_payload_version TEXT
+ha_pri_engine_version TEXT
+```
+
+Recommended operational fields:
+
+```txt
+ha_pri_verification_status TEXT
+ha_pri_verified_at TEXT
+ha_pri_backfill_status TEXT
+```
+
+Likely unnecessary as separate stored fields:
+
+```txt
+ha_pri_adapter
+ha_pri_published_at
+ha_pri_retrieved_at
+```
+
+Reason: adapter, published timestamp, and retrieved/scraped timestamp already exist or should exist as first-class row fields. Duplicating them inside Ha-Pri-specific columns risks drift. The signing payload should read those canonical row fields directly during verification.
+
+Minimum practical schema:
+
+```sql
+ALTER TABLE scrape_results ADD COLUMN ha_pri_sig_v2 TEXT;
+ALTER TABLE scrape_results ADD COLUMN ha_pri_canonical_content_sha256 TEXT;
+ALTER TABLE scrape_results ADD COLUMN ha_pri_semantic_fingerprint_sha256 TEXT;
+ALTER TABLE scrape_results ADD COLUMN ha_pri_signing_payload_version TEXT;
+ALTER TABLE scrape_results ADD COLUMN ha_pri_engine_version TEXT;
+ALTER TABLE scrape_results ADD COLUMN ha_pri_verification_status TEXT;
+ALTER TABLE scrape_results ADD COLUMN ha_pri_verified_at TEXT;
+ALTER TABLE scrape_results ADD COLUMN ha_pri_backfill_status TEXT;
+```
+
+Storage status values should be boring and explicit:
+
+```txt
+valid
+invalid
+unknown
+not_checked
+```
+
+Backfill status values should avoid pretending old rows were originally v2-stamped:
+
+```txt
+none
+backfilled
+unknown_origin
+failed
+```
+
+## 4. Write-Path Design
+
+The future Worker write path should dual-stamp v1 and v2 for new rows.
+
+Recommended write sequence:
+
+1. Adapter returns raw candidate content.
+2. Existing Worker scoring computes current DAR fields and Ha-Pri v1.
+3. Write path prepares Ha-Pri v2 input:
+   - `resultId`: row id that will be stored
+   - `rawContent`: canonical row raw content
+   - `semanticFingerprint`: semantic fingerprint material or stored fingerprint
+   - `adapter`: adapter id
+   - `publishedAt`: normalized source publication timestamp or `null`
+   - `retrievedAt`: normalized scrape/retrieval timestamp
+   - `engineVersion`: FreshContext engine/package version or explicit Worker engine version
+4. If required v2 material is complete, calculate:
+   - canonical content SHA-256
+   - semantic fingerprint SHA-256
+   - signing payload version
+   - Ha-Pri v2 signature
+5. Store v1 fields as today.
+6. Store v2 fields alongside v1 fields.
+7. If material is incomplete, store unknown-compatible metadata and do not pretend the row is valid.
+
+Recommended incomplete-material behavior:
+
+```txt
+ha_pri_sig_v2 = null
+ha_pri_verification_status = "unknown"
+ha_pri_backfill_status = "none"
+```
+
+Canonical content should be produced by the same pure helper behavior used in Core golden vectors. Do not invent a second Worker-only canonicalization contract.
+
+Semantic fingerprint should be produced before signing and should be stable across retries for the same underlying source item. If the fingerprint is missing, v2 signing should fall back to `unknown`, not a fake valid signature.
+
+Engine version should be explicit. The safest initial choice is the package/server version used by the running Worker build.
+
+## 5. Read / Debug Verification Design
+
+Future read verification should be a pure recomputation:
+
+```txt
+verifyHaPriV2(row) -> valid | invalid | unknown
+```
+
+Suggested internal verification input:
+
+```ts
+{
+  resultId: row.id,
+  rawContent: row.raw_content,
+  semanticFingerprint: row.semantic_fingerprint,
+  adapter: row.adapter,
+  publishedAt: row.published_at,
+  retrievedAt: row.scraped_at,
+  engineVersion: row.ha_pri_engine_version
+}
+```
+
+Debug output may include:
+
+```json
+{
+  "ha_pri_v2": {
+    "status": "valid",
+    "checked_at": "2026-06-11T12:00:00.000Z",
+    "payload_version": "FRESHCONTEXT_HA_PRI_V2",
+    "canonical_content_sha256": "sha256...",
+    "semantic_fingerprint_sha256": "sha256..."
+  }
+}
+```
+
+Safe public output should be smaller:
+
+```json
+{
+  "provenance": {
+    "ha_pri_v2_status": "valid"
+  }
+}
+```
+
+Do not expose signing payloads or debug hashes in broad public outputs unless there is a clear user need.
+
+Suggested staged behavior:
+
+- Phase 1: report-only verification in internal/debug output.
+- Phase 2: warning in read/debug path when invalid.
+- Phase 3: optional strict mode for private deployments.
+- Phase 4: possible reject/block policy only after replay data and operational evidence.
+
+Invalid should not become automatic rejection first. A migration bug, canonicalization mismatch, or schema rollout issue could otherwise hide useful rows during rollout.
+
+## 6. Compatibility And Backfill
+
+Old rows must remain readable.
+
+Compatibility rules:
+
+- v1-only rows verify as `unknown` for v2.
+- Rows with missing v2 fields verify as `unknown`.
+- Rows with malformed v2 signatures verify as `invalid`.
+- Rows with present but mismatched v2 signatures verify as `invalid`.
+- Missing `ha_pri_sig_v2` is not the same as tampering.
+- Existing `ha_pri_sig` remains readable for historical continuity.
+
+Backfill rules:
+
+- Backfilled provenance must be marked as `backfilled` or `unknown_origin`.
+- Backfill must not imply the row was v2-stamped at original write time.
+- Backfill should preserve original row timestamps.
+- Backfill should record when verification/backfill happened.
+- Backfill should be reversible or repeatable where practical.
+
+Possible backfill process:
+
+1. Select rows missing `ha_pri_sig_v2`.
+2. Reconstruct v2 input from stored row fields.
+3. If required material is complete, calculate v2 fields.
+4. Store v2 fields with `ha_pri_backfill_status = "backfilled"`.
+5. If required material is incomplete, store `ha_pri_verification_status = "unknown"` and `ha_pri_backfill_status = "unknown_origin"`.
+6. Report counts for backfilled, unknown, invalid, and failed rows.
+
+## 7. Security Boundary
+
+Plain SHA-256 gives deterministic integrity and audit checks.
+
+Plain SHA-256 does not prove private origin authentication. Anyone with all payload fields can recompute a plain SHA-256 signature.
+
+Ha-Pri v2 helps detect:
+
+- accidental row corruption
+- changed content after write
+- changed semantic fingerprint material
+- changed adapter/timestamp/version fields included in the signing payload
+- malformed stored signatures
+
+Ha-Pri v2 does not solve:
+
+- truth certification
+- legal, medical, tax, employment, academic, or investment correctness
+- private origin authentication without a secret or private key
+- compromise of the write path before signing
+- compromise of all row fields plus signature under plain SHA-256
+
+Recommendation:
+
+Do not add HMAC/private signing immediately to the open package. Keep the open package deterministic and stateless.
+
+Consider HMAC or private-key signing later for:
+
+- hosted FreshContext endpoints
+- private production deployments
+- paid/tenant-specific infrastructure
+- environments where the verifier must know the row was stamped by a trusted FreshContext deployment
+
+If HMAC/private signing is added later, it requires:
+
+- secret storage outside the repo
+- key ids
+- key rotation
+- old-key verification policy
+- signer/verifier boundary documentation
+- tests proving secrets are never logged or returned
+
+## 8. Threat Model
+
+Threats considered:
+
+### Accidental row corruption
+
+Ha-Pri v2 helps by recomputing the expected signature and surfacing `invalid`.
+
+### Stale or partial provenance
+
+Ha-Pri v2 helps by returning `unknown` when required material is missing. The system should not pretend such rows are valid.
+
+### Tampered D1 rows
+
+Ha-Pri v2 helps if an attacker changes stored content or bound fields without also updating all matching v2 fields.
+
+Plain SHA-256 does not help if an attacker can rewrite all row fields and recompute the public signature.
+
+### Malformed signatures
+
+Malformed, blank, or nonmatching signatures should produce `invalid` or `unknown` according to current helper behavior. They should not crash reads.
+
+### Recomputed public SHA-256 signatures
+
+Because v2 currently uses plain SHA-256, a party with all fields can recompute a matching signature. HMAC/private signing is the later answer if origin authentication becomes necessary.
+
+### Debug endpoint leakage
+
+Debug routes should remain authenticated. Public outputs should avoid exposing full signing payloads or internal row material unless deliberately needed.
+
+### Secret exposure if HMAC is added later
+
+HMAC/private signing introduces secret-management risk. Secrets must live in deployment configuration, never in docs, fixtures, npm package output, or client-visible responses.
+
+## 9. Tests Needed Before Implementation
+
+Future implementation should add tests before production rollout:
+
+- D1 migration tests if the migration harness supports them.
+- Write-path stamping tests for new rows.
+- Dual-stamp tests proving v1 remains unchanged.
+- Read-path verification tests for `valid`, `invalid`, and `unknown`.
+- Old-row tests proving missing v2 fields are `unknown`, not invalid.
+- Tampered-row tests for changed content, semantic fingerprint, adapter, timestamps, and engine version.
+- Malformed signature tests.
+- Debug output safety tests.
+- Public output minimization tests.
+- Backfill tests with complete and incomplete rows.
+- Worker dry-run validation.
+- HMAC/private signing tests if that later lands.
+
+Do not add fake production-enforcement tests before implementation exists.
+
+## 10. Rollout Phases
+
+### Phase 0: Core helper and golden vectors
+
+Done.
+
+Includes:
+
+- pure Core Ha-Pri v2 helper
+- deterministic golden vectors
+- valid / invalid / unknown verification behavior
+
+### Phase 1: Storage schema design
+
+This document.
+
+No migration yet.
+
+### Phase 2: Write-path dual-stamp v1 + v2
+
+Add D1 columns and write v2 fields for new rows only. Keep v1 intact.
+
+### Phase 3: Report-only read/debug verification
+
+Recompute v2 on read/debug paths and report status. Do not reject rows yet.
+
+### Phase 4: Backfill tooling
+
+Backfill historical rows only with explicit `backfilled` or `unknown_origin` markers.
+
+### Phase 5: Optional strict mode
+
+Private deployments may opt into warnings or blocking for invalid rows after enough evidence.
+
+### Phase 6: Hosted/private HMAC signing
+
+Add origin-authenticated signing only if hosted/private use cases require it.
+
+## 11. Non-Goals
+
+This pass does not implement:
+
+- D1 migration
+- Worker enforcement
+- read-path rejection
+- debug endpoint changes
+- HMAC/private signing
+- backfill script
+- npm publish
+- version bump
+- Cloudflare deploy
+- public security claim upgrade
+- new MCP tools
+- ranking/scoring changes
+- Operator/retrieve behavior
+
+## Release Gates For Future Implementation
+
+Before any implementation pass can claim production enforcement:
+
+- migrations must be reviewed and dry-run
+- new rows must dual-stamp v1 and v2
+- v1 compatibility tests must pass
+- v2 read verification tests must pass
+- old-row `unknown` behavior must be proven
+- tampered-row `invalid` behavior must be proven
+- debug output must avoid leaking secrets or excessive internals
+- Worker dry-run must pass
+- Trust gate must pass with effective fail 0
+- public docs must say exactly what is implemented
+
+## Product Interpretation
+
+Ha-Pri v2 is a provenance hardening lane for stored FreshContext rows. It supports the larger product story only when it stays honest:
+
+```txt
+candidate context in
+decision-ready context out
+optional provenance verification for stored rows
+```
+
+It is not a truth engine and not a substitute for authentication, authorization, or hosted tenant isolation.

package/docs/RELEASE_INTEGRITY.md

@@ -22,7 +22,7 @@ This document describes release hardening practices for future FreshContext pack
   - Confirm fresh consumer `npm audit --omit=dev` is clean.
 - Run a stale-claim scan across public docs and package-facing files.
 - Run a secret scan before sharing archives, diligence folders, or package artifacts.
-- Keep operational demo runbooks, buyer scripts, outreach plans, diligence checklists, and negotiation materials outside the public npm package.
+- Keep operational demo runbooks, buyer scripts, outreach plans, diligence checklists, and private commercial materials outside the public npm package.
 
 ## Package Exclusion Checks
 

package/docs/SIGNAL_CONTRACT.md

@@ -1,8 +1,30 @@
-# FreshContext Signal Contract v1
-
-FreshContext Signal Contract v1 defines the Core shape for a retrieved signal before it is ranked, wrapped, stored, or passed to an agent workflow.
-
-It is an additive Core API. It does not change MCP tool schemas, Worker runtime behavior, D1 schema, Store scoring, feeds, or deployment behavior.
+# FreshContext Signal Contract v1
+
+FreshContext Signal Contract v1 is the current FreshContext input standard: the stable shape for candidate context that should be judged before it reaches a model.
+
+In plain terms:
+
+```text
+candidate context -> Signal Contract v1 -> FreshContext judgment -> decision-ready context
+```
+
+The Signal Contract is live product architecture. It is used by Core normalization, `evaluate_context`, bring-your-own-context demos, reference adapter signal paths, and future batch validation.
+
+Do not rename this with future-signal terminology. Future context signals, control signals, provenance confidence signals, and richer decision metadata are optional future layers on top of this stable input shape. They are not replacements for Signal Contract v1 and are not required fields today.
+
+It is an additive Core API. It does not change MCP tool schemas, Worker runtime behavior, D1 schema, Store scoring, feeds, or deployment behavior.
+
+## Batch Validation Harness
+
+Source checkouts include a small validation harness for testing larger Signal Contract v1 batches:
+
+```bash
+npm run batch:validate -- examples/batches/signal-contract-v1.academic.json
+```
+
+The harness reads caller-provided JSON, evaluates it through Core, summarizes status/date/decision counts, and prints a structured evidence block. It does not add required fields to the Signal Contract and does not fetch, crawl, scan folders, or call reference adapters.
+
+Replay output includes concise decision explanations and small reason-code lists for top results and human-review mismatches. These are reporting aids for auditability: they surface whether relevance, freshness, date confidence, status, utility, score normalization, or Source Profile behavior influenced a treatment label. They do not change Core scoring, ranking, normalization, or decision thresholds, and they do not certify truth.
 
 ## Contract Version
 
@@ -16,9 +38,9 @@ Every normalized signal includes:
 contract_version: "freshcontext.signal.v1"
 ```
 
-## Input Shape
-
-`FreshContextSignalInput` accepts the common fields used by adapters, agents, ranking, and future Store wiring:
+## Input Shape
+
+`FreshContextSignalInput` accepts the common fields used by adapters, agents, ranking, `evaluate_context`, and future Store wiring:
 
 ```ts
 interface FreshContextSignalInput {
@@ -38,7 +60,21 @@ interface FreshContextSignalInput {
 }
 ```
 
-`published_at` is the canonical signal timestamp. `content_date` is accepted as an adapter/envelope compatibility alias.
+`published_at` is the canonical signal timestamp. `content_date` is accepted as an adapter/envelope compatibility alias.
+
+Minimal caller-provided input usually looks like:
+
+```json
+{
+  "title": "Example source",
+  "content": "Candidate context text...",
+  "source": "https://example.com/source",
+  "source_type": "official_docs",
+  "published_at": "2026-06-01T00:00:00.000Z",
+  "retrieved_at": "2026-06-09T00:00:00.000Z",
+  "semantic_score": 0.92
+}
+```
 
 ## Normalized Output
 
@@ -62,16 +98,166 @@ interface FreshContextSignal {
 }
 ```
 
-## Normalization Rules
-
-- Missing or invalid `published_at` / `content_date` becomes `published_at: null`.
+## Normalization Rules
+
+- Missing or invalid `published_at` / `content_date` becomes `published_at: null`.
 - `content_date` maps to `published_at` when `published_at` is absent.
 - Meaningfully future-dated timestamps are cleared and receive `date_confidence: "unknown"`.
 - Small clock skew is tolerated by the same Core freshness policy used by envelope scoring.
 - Failed, empty, timeout, blocked, or error-looking content becomes `status: "failed"`.
 - Missing, invalid, negative, or oversized `semantic_score` is clamped into `0..1`.
-- `metadata` is shallow-copied so normalization does not mutate caller-owned objects.
-- `reasons` records meaningful normalization changes.
+- `metadata` is shallow-copied so normalization does not mutate caller-owned objects.
+- `reasons` records meaningful normalization changes.
+
+## Examples
+
+These examples are intentionally small. They show the current contract shape, not future optional metadata.
+
+### Valid Candidate Signals
+
+Academic research:
+
+```json
+{
+  "title": "A fresh retrieval-augmented generation benchmark",
+  "content": "The paper reports a 2026 benchmark for retrieval-augmented generation systems.",
+  "source": "https://arxiv.org/abs/2606.00001",
+  "source_type": "arxiv",
+  "published_at": "2026-06-01T09:00:00.000Z",
+  "retrieved_at": "2026-06-09T12:00:00.000Z",
+  "semantic_score": 0.94,
+  "metadata": {
+    "profile": "academic_research"
+  }
+}
+```
+
+Official docs:
+
+```json
+{
+  "title": "API changelog",
+  "content": "The official changelog documents the current API behavior.",
+  "source": "https://docs.example.com/changelog",
+  "source_type": "official_docs",
+  "published_at": "2026-06-08T10:00:00.000Z",
+  "retrieved_at": "2026-06-09T12:00:00.000Z",
+  "semantic_score": 0.88
+}
+```
+
+Jobs/opportunities:
+
+```json
+{
+  "title": "AI tools engineer",
+  "content": "A current remote role for an AI tools engineer.",
+  "source": "https://jobs.example.com/ai-tools-engineer",
+  "source_type": "jobs",
+  "published_at": "2026-06-07T08:00:00.000Z",
+  "retrieved_at": "2026-06-09T12:00:00.000Z",
+  "semantic_score": 0.86
+}
+```
+
+Market/finance:
+
+```json
+{
+  "title": "Company quarterly update",
+  "content": "The company reported current quarter revenue and guidance.",
+  "source": "https://investors.example.com/q2-update",
+  "source_type": "finance",
+  "published_at": "2026-06-09T07:00:00.000Z",
+  "retrieved_at": "2026-06-09T12:00:00.000Z",
+  "semantic_score": 0.83
+}
+```
+
+Social pulse:
+
+```json
+{
+  "title": "Developer discussion",
+  "content": "Developers are discussing setup friction and recent adoption.",
+  "source": "https://news.ycombinator.com/item?id=123456",
+  "source_type": "hackernews",
+  "published_at": "2026-06-09T11:00:00.000Z",
+  "retrieved_at": "2026-06-09T12:00:00.000Z",
+  "semantic_score": 0.71
+}
+```
+
+### Invalid Or Risky Candidate Signals
+
+Missing date:
+
+```json
+{
+  "title": "Relevant source with no date",
+  "content": "Useful candidate context, but no publication timestamp is available.",
+  "source": "https://example.com/no-date",
+  "source_type": "official_docs",
+  "retrieved_at": "2026-06-09T12:00:00.000Z",
+  "semantic_score": 0.78
+}
+```
+
+Invalid timestamp:
+
+```json
+{
+  "title": "Invalid date source",
+  "content": "Candidate context with malformed date metadata.",
+  "source": "https://example.com/bad-date",
+  "source_type": "official_docs",
+  "published_at": "not-a-date",
+  "retrieved_at": "2026-06-09T12:00:00.000Z",
+  "semantic_score": 0.78
+}
+```
+
+Meaningfully future-dated timestamp:
+
+```json
+{
+  "title": "Future-dated source",
+  "content": "Candidate context whose publication timestamp is after retrieval time.",
+  "source": "https://example.com/future-date",
+  "source_type": "official_docs",
+  "published_at": "2026-06-09T12:06:00.000Z",
+  "retrieved_at": "2026-06-09T12:00:00.000Z",
+  "semantic_score": 0.78
+}
+```
+
+Failed/error-looking content:
+
+```json
+{
+  "title": "Blocked source",
+  "content": "[Error] upstream timeout",
+  "source": "https://example.com/blocked",
+  "source_type": "official_docs",
+  "published_at": "2026-06-09T10:00:00.000Z",
+  "retrieved_at": "2026-06-09T12:00:00.000Z",
+  "semantic_score": 0.91
+}
+```
+
+Out-of-range semantic score:
+
+```json
+{
+  "title": "Overscored source",
+  "content": "Candidate context with an out-of-range relevance score.",
+  "source": "https://example.com/overscored",
+  "source_type": "official_docs",
+  "published_at": "2026-06-09T10:00:00.000Z",
+  "retrieved_at": "2026-06-09T12:00:00.000Z",
+  "semantic_score": 1.7
+}
+```
 
 ## Relationship to Existing Core Types
 
@@ -84,6 +270,16 @@ The signal contract does not replace existing Core types:
 
 The contract gives these surfaces a shared signal vocabulary without requiring Store, Worker, or MCP schema changes.
 
-## Boundary
-
-Signal Contract v1 does not determine truth, certify data, or provide legal, medical, tax, or financial advice. It provides normalized context metadata for freshness, provenance, relevance, and workflow review.
+## Boundary
+
+Signal Contract v1 does not determine truth, certify data, or provide legal, medical, tax, or financial advice. It provides normalized context metadata for freshness, provenance, relevance, and workflow review.
+
+## Future Metadata Boundary
+
+Future context signals, control signals, ingestion quality signals, structure preservation signals, and provenance confidence signals may later improve decisions such as `cite_as_primary`, `needs_refresh`, `needs_verification`, or `exclude`.
+
+Those are roadmap metadata layers. They should remain optional until tests prove they improve decisions. The public input contract should stay boring and stable:
+
+```text
+title + content + source + source_type + published_at + retrieved_at + semantic_score
+```