freshcontext-mcp 0.3.16 → 0.3.18

package/docs/CORE_API.md

@@ -0,0 +1,224 @@
+# FreshContext Core API
+
+FreshContext Core is the reusable engine layer in the current integrated MCP/Core package. It owns signal normalization, envelope creation, freshness scoring, failure honesty, rank/explain primitives, the context-utility primitive, and pure provenance helpers.
+
+MCP, Worker HTTP, future REST, and future CLI/SDK surfaces should use Core as the contract center instead of redefining freshness or envelope behavior per host.
+
+## Stable Public Core API
+
+Import stable Core functions from:
+
+```ts
+import {
+  calculateFreshnessScore,
+  formatForLLM,
+  looksLikeFailedAdapterContent,
+  scoreLabel,
+  stampFreshness,
+  toStructuredJSON,
+} from "./src/core/index.js";
+```
+
+### Envelope
+
+- `stampFreshness(result, options, adapter)` creates a `FreshContext` object from adapter output.
+- `formatForLLM(ctx, options?)` renders the text envelope and trailing structured JSON block.
+- `toStructuredJSON(ctx)` returns the machine-readable FreshContext JSON shape.
+
+### Scoring
+
+- `calculateFreshnessScore(content_date, retrieved_at, adapter)` returns a freshness score from `0..100`, or `null` when the content date cannot be trusted.
+- `scoreLabel(score)` maps a numeric freshness score to a human-readable label.
+
+### Guards
+
+- `looksLikeFailedAdapterContent(raw)` detects empty, security, timeout, and error-like adapter output so failed content is not stamped as fresh high-confidence context.
+
+## Core Evaluation Pipeline
+
+The Core evaluation pipeline is the pure orchestration layer over the existing primitives.
+
+Public exports:
+
+- `evaluateSignal(input, options?)`
+- `evaluateSignals(inputs, options?)`
+- `CoreSignalEvaluationOptions`
+- `CoreSignalEvaluationResult`
+- `CoreSignalEnvelopeResult`
+- `CoreSignalProvenanceOptions`
+
+`evaluateSignal` normalizes a signal, applies timestamp/failure guards, computes `freshness_score`, computes context-conditioned utility, ranks/explains the signal, optionally creates a FreshContext envelope, and optionally prepares Ha-Pri v2 provenance material.
+
+It does not fetch, cache, write D1, inspect Worker bindings, know MCP tool schemas, deploy, or publish. Hosts decide whether to store, cache, transmit, or expose the returned result.
+
+`evaluateSignals` evaluates each input and returns evaluations sorted by existing `rankSignal` final score, preserving input order when scores tie. Context utility is returned as a sidecar and does not replace `final_score`.
+
+Context utility is returned as sidecar output in the current pipeline; it does not replace or modify the default `rankSignal` / `evaluateSignals` ordering. A future pass may add an explicit utility-weighted ranking mode.
+
+Local demo:
+
+```bash
+npm run demo:evaluate:file -- examples/sources.academic.example.json
+npm run demo:evaluate:file -- examples/sources.jobs.example.json
+```
+
+The demo reads caller-provided JSON with `profile`, `intent`, and `signals`, then returns decision-first output. It does not fetch URLs, crawl, read folders, deploy REST, or implement Operator mode.
+
+### Stable Types
+
+- `FreshContext`
+- `AdapterResult`
+- `ExtractOptions`
+- `EnvelopeFormatOptions`
+- `SignalConfidence`
+
+These types describe the stable envelope and adapter result contract.
+
+## Signal Contract v1
+
+Signal Contract v1 is the additive Core shape for a retrieved signal before it is ranked, wrapped, stored, or passed to an agent workflow.
+
+Public exports:
+
+- `SIGNAL_CONTRACT_VERSION`
+- `normalizeSignal(input, options?)`
+- `FreshContextSignalInput`
+- `FreshContextSignal`
+- `SignalDateConfidence`
+- `SignalContractVersion`
+- `SignalNormalizeOptions`
+
+`published_at` is the canonical signal timestamp. `content_date` is accepted as an adapter/envelope compatibility alias. Normalization clears invalid or meaningfully future-dated timestamps, marks failed/error-looking content as `status: "failed"`, clamps `semantic_score` into `0..1`, and records normalization reasons.
+
+See [Signal Contract v1](./SIGNAL_CONTRACT.md).
+
+## Source Profiles
+
+Source Profiles are early public Core metadata for describing how classes of information age, fail, rank, and explain.
+
+Public exports:
+
+- `BUILT_IN_SOURCE_PROFILES`
+- `getSourceProfile(profileId)`
+- `listSourceProfiles()`
+- `SourceProfile`
+- `SourceProfileId`
+- `SourceAuthorityHint`
+- `SourceDatePolicy`
+- `SourceFailurePolicy`
+- `SourceSurface`
+
+They reframe the 21 MCP tools as reference adapters and source-profile examples instead of the product identity. They do not implement `retrieve(...)`, Operator mode, adapter selection, crawling, local file search, or any host/runtime behavior.
+
+## Decision Helper
+
+The decision helper translates a Core evaluation result into user-facing action meaning.
+
+Public exports:
+
+- `interpretEvaluation(evaluation, options?)`
+- `interpretEvaluations(evaluations, options?)`
+- `ContextDecision`
+- `IntentProfileId`
+- `ContextDecisionOptions`
+- `ContextDecisionResult`
+
+Supported decisions:
+
+- `use_first`
+- `cite_as_primary`
+- `cite_as_supporting`
+- `use_as_background`
+- `needs_verification`
+- `needs_refresh`
+- `watch_only`
+- `exclude`
+
+Supported intent profiles:
+
+- `citation_check`
+- `student_research`
+- `developer_adoption`
+- `job_search`
+- `market_watch`
+- `business_due_diligence`
+- `medical_literature_triage`
+
+The helper consumes existing `CoreSignalEvaluationResult` fields plus optional Source Profile metadata. It does not change `evaluateSignal`, `evaluateSignals`, `rankSignal`, ranking order, freshness math, utility math, envelopes, provenance, or host behavior.
+
+FreshContext decisions judge citation readiness, context usefulness, freshness, traceability, and uncertainty. They do not certify truth or provide legal, medical, tax, employment, academic, or investment advice.
+
+Demo output will be updated separately so presentation stays separate from Core decision logic.
+
+## Public Ranking Primitives
+
+The ranking primitives are public, but consumers should treat their score scales carefully:
+
+- `rankSignal(signal, options?)`
+- `rankSignals(signals, options?)`
+- `explainSignal(rankedSignalLike)`
+- `FreshSignal`
+- `RankedSignal`
+- `RankOptions`
+
+Score scales:
+
+- `semantic_score`: normalized `0..1`
+- `final_score`: normalized `0..1`
+- `freshness_score`: FreshContext freshness score `0..100`, or `null`
+
+Ranking combines semantic relevance and freshness into a deterministic order. It does not own retrieval, embedding, vector search, storage, or host-specific scoring policy.
+
+## Experimental Utility Primitive
+
+The context-conditioned utility primitive is pure and tested, but it is not production-wired into MCP ranking, Worker feeds, Store scoring, or runtime behavior.
+
+Experimental exports:
+
+- `calculateContextUtility`
+- `ContextUtilityStatus`
+- `ContextUtilityInput`
+- `ContextUtilityResult`
+
+These are pure Core math. They are now connected inside `evaluateSignal` as sidecar utility output, but they are not production-wired into MCP ranking, Worker feeds, Store scoring, or runtime behavior.
+
+## Provenance Helpers
+
+Ha-Pri v2 is available as pure Core helper functionality:
+
+- `canonicalizeHaPriContent`
+- `sha256Hex`
+- `calculateHaPriV2`
+- `verifyHaPriV2`
+- `HaPriV2Input`
+- `HaPriV2Result`
+- `HaPriV2VerificationResult`
+
+`evaluateSignal` can optionally prepare Ha-Pri v2 material when `includeProvenance` is set and required input material is present. Core does not persist provenance, add D1 columns, verify rows on read, reject rows, or replace Worker Ha-Pri v1 behavior.
+
+## Internal, Policy, and Compatibility Exports
+
+- `clampScore` is an internal ranking helper. It is currently exported for tests and utility use, but it should not be presented as a primary buyer-facing API.
+- `LAMBDA` is the current policy constant table used by freshness scoring. It documents the reference decay policy, but it is not a buyer-facing tuning API.
+
+Compatibility lanes should remain:
+
+- `src/types.ts` re-exports legacy adapter types from Core.
+- `src/tools/freshnessStamp.ts` re-exports envelope helpers for older MCP/npm import paths.
+
+These lanes protect existing imports while Core becomes the center. Do not remove them until downstream imports have been migrated intentionally.
+
+## What Core Does Not Own
+
+Core does not own:
+
+- MCP transport
+- Cloudflare runtime behavior
+- KV cache policy
+- Cache metadata injection
+- D1, feed, or cron behavior
+- Store/feed scoring and provenance persistence
+- Hosted dashboard, API, deployment, or runtime concerns
+
+Hosts may wrap Core outputs with their own transport, cache, session, rate-limit, or persistence metadata, but they should not fork the Core envelope and freshness contract without an explicit compatibility reason.
+

package/docs/DEPENDENCY_DILIGENCE.md

@@ -0,0 +1,63 @@
+# FreshContext Dependency Diligence Notes
+
+This document records dependency and license diligence notes from the Trust L4/L5 cleanup. It is not legal advice and does not replace professional review for external review, distribution, or formal diligence.
+
+## Current Audit Status
+
+As of Pass 8-AB:
+
+- `npm audit --omit=dev`: clean.
+- `npm audit`: clean.
+- The published MCP npm package excludes the Apify Actor entrypoint and does not install Apify/Crawlee in normal consumer installs.
+- The previous moderate `qs` and `ws` advisories were resolved with narrow transitive overrides in the source checkout.
+- No package version change was made.
+
+## Resolved Advisories
+
+`qs`
+
+- Previous severity: moderate.
+- Path: `@modelcontextprotocol/sdk -> express/body-parser -> qs`.
+- Resolution: pinned through npm `overrides` to `qs@6.15.2`.
+
+`ws`
+
+- Previous severity: moderate.
+- Historical source-checkout path: `apify -> ws`.
+- Resolution: pinned through npm `overrides` to `ws@8.20.1` for source-checkout Apify Actor workflows.
+
+`file-type`
+
+- Previous severity: moderate in fresh consumer installs.
+- Historical consumer path: `apify -> @crawlee/utils -> file-type`.
+- Resolution: Apify/Crawlee were removed from the normal published MCP package dependency surface. Apify remains a source-checkout / separate-actor concern.
+
+## License Inventory Notes
+
+The Trust L4 license inventory was broadly permissive, including MIT, Apache-2.0, BSD variants, ISC, 0BSD, BlueOak-1.0.0, and similar permissive variants.
+
+No GPL, AGPL, LGPL, MPL, EPL, CDDL, or similar copyleft licenses were reported in the Trust L4 scan.
+
+`map-stream@0.1.0`
+
+- Scanner result: `UNKNOWN`.
+- Path observed during L4: transitive through source-checkout `apify` / Crawlee-related dependencies.
+- Diligence note: package metadata appears incomplete, but the installed package includes an MIT-style license file.
+- Action: keep as a source-checkout / actor-packaging diligence note and recheck before external diligence or Apify Actor distribution.
+
+`caniuse-lite`
+
+- Scanner result: `CC-BY-4.0`.
+- Diligence note: preserve and review attribution requirements before external diligence, bundled distribution, or a formal review package.
+
+## Before External Diligence or Distribution
+
+- Rerun `npm audit --omit=dev`.
+- Rerun `npm audit`.
+- Rerun dependency license inventory.
+- Review scanner-unknown packages.
+- Review `caniuse-lite` attribution requirements.
+- Generate an SBOM if requested by an evaluator, reviewer, or downstream distributor.
+- Review dependency and license posture with qualified counsel when external distribution or formal diligence requires it.
+
+Do not treat this document as a legal conclusion.

package/docs/HA_PRI_V2_DESIGN.md

@@ -0,0 +1,279 @@
+# Ha-Pri v2 Design
+
+Status: design + pure Core helper
+Phase: Math Spine Phase 3-A / 3-B
+Runtime impact: none
+
+## Purpose
+
+Ha-Pri v2 is an additive provenance-hardening model for FreshContext Store/Ledger rows.
+
+The goal is to keep Ha-Pri v1 readable while designing a stronger future signature that binds a row to canonical content, semantic identity, source metadata, timestamps, and engine version.
+
+Phase 3-B adds pure Core helper functions and deterministic tests for the v2 model. Phase 3-C adds `examples/ha-pri-v2-example.ts`, a deterministic developer fixture showing `calculateHaPriV2` and `verifyHaPriV2` returning valid, invalid, and unknown verification states. Production Store wiring remains future work. This document does not change the D1 schema, change Worker write paths, migrate old rows, add HMAC secrets, or alter production scoring.
+
+## Current Ha-Pri v1 Audit
+
+Ha-Pri v1 is implemented today as a provenance stamp and audit reference, not yet hard tamper enforcement.
+
+### Where v1 Lives
+
+Current implementation points:
+
+- `worker/src/intelligence.ts`
+  - `PROVENANCE_SALT = "FRESHCONTEXT_DAR_V1"`
+  - `generateAuditSig(resultId, contentHash)`
+  - `scoreSignal(...)` computes `ha_pri_sig`
+- `worker/src/worker.ts`
+  - migration adds `ha_pri_sig TEXT`
+  - cron write path stores `ha_pri_sig` in `scrape_results`
+  - `/v1/intel/feed/:profile_id` returns `ha_pri_sig` in `intelligence_stamps`
+- `tests/mathSpine.test.ts`
+  - checks that `generateAuditSig` matches the documented v1 formula
+
+### v1 Formula
+
+```text
+ha_pri_sig = SHA-256(
+  result_id + ":" +
+  content_hash + ":" +
+  "FRESHCONTEXT_DAR_V1"
+)
+```
+
+### What v1 Binds
+
+Ha-Pri v1 binds:
+
+- the generated `result_id`
+- the current `content_hash` argument passed into `scoreSignal`
+- the engine/version salt string `FRESHCONTEXT_DAR_V1`
+
+In the current Worker cron path, `content_hash` is the value named `result_hash`, produced by `simpleHash(raw)`.
+
+### Current Hash Input
+
+The current `result_hash` is a small rolling hash:
+
+```ts
+let h = 0;
+for (let i = 0; i < str.length; i++) h = Math.imul(31, h) + str.charCodeAt(i) | 0;
+return Math.abs(h).toString(36);
+```
+
+This is useful for cheap change detection, but it is not a cryptographic content digest.
+
+### Storage and Output
+
+Ha-Pri v1 is stored in D1:
+
+- `scrape_results.ha_pri_sig`
+
+It is returned through the live intelligence feed:
+
+- `signals[].intelligence_stamps.ha_pri_sig`
+
+### Verification Status
+
+Current v1 behavior:
+
+- generated on write: yes
+- stored in D1: yes
+- returned in feed/API output: yes
+- recomputed on read: no
+- used to reject tampered rows: no
+- tied to canonical raw content SHA-256: no
+
+So Ha-Pri v1 works as a provenance stamp and audit reference. It does not yet work as hard tamper enforcement.
+
+## Weaknesses in v1
+
+1. The signature uses a weak content-hash input.
+
+   `ha_pri_sig` is SHA-256, but it currently binds to the rolling `result_hash`, not to canonical raw content bytes. The v1 signature inherits the collision risk and ambiguity of the weaker input.
+
+2. No read-time verification exists.
+
+   Feed and debug reads return the stored signature, but they do not recompute it and compare stored vs recomputed values.
+
+3. No canonicalization contract exists for signed content.
+
+   The current signature signs a hash value, not a documented canonical representation of the row content.
+
+4. v1 does not bind all fields needed for provenance.
+
+   It does not directly bind adapter, published timestamp, scraped timestamp, semantic fingerprint, or a schema marker beyond the fixed salt.
+
+5. v1 is not authentication.
+
+   The salt is public. Anyone with row fields can compute the v1 signature. That is acceptable for a provenance reference, but it should not be presented as proof of origin from a private signing authority.
+
+## Ha-Pri v2 Design Goals
+
+Ha-Pri v2 should be:
+
+- additive, not a breaking migration
+- deterministic
+- recomputable
+- explicit about canonicalization
+- stronger than v1 for content integrity
+- safe to run without secrets
+- compatible with future HMAC signing, without requiring it now
+- clear about verification status: valid, invalid, or unknown
+
+## Proposed v2 Fields
+
+Future Store/Ledger rows may add:
+
+```text
+canonical_content_sha256 TEXT
+semantic_fingerprint_sha256 TEXT
+ha_pri_sig_v2 TEXT
+ha_pri_v2_status TEXT
+ha_pri_v2_checked_at TEXT
+```
+
+These are design-level names only. No schema change is made in this phase.
+
+### canonical_content_sha256
+
+`canonical_content_sha256` is:
+
+```text
+SHA-256(canonical raw content)
+```
+
+It binds the actual content after deterministic normalization.
+
+### semantic_fingerprint_sha256
+
+`semantic_fingerprint_sha256` is:
+
+```text
+SHA-256(normalized title + canonical URL + publication date)
+```
+
+It is a full SHA-256 version of the current shorter semantic fingerprint idea.
+
+## Canonicalization Rules
+
+All canonicalization should be deterministic.
+
+Recommended rules:
+
+1. Use UTF-8.
+2. Normalize line endings to `\n`.
+3. Trim trailing whitespace on each line.
+4. Preserve meaningful internal whitespace.
+5. Normalize null or missing optional fields to the literal string `"null"`.
+6. Use stable field order.
+7. Use ISO-8601 timestamps where available.
+8. Do not include fields whose values change during read-time verification unless they are explicitly part of the signed record.
+9. Version the canonicalization contract.
+
+For future implementation, canonicalization should live in a pure helper with deterministic fixtures.
+
+## Proposed v2 Formula
+
+```text
+ha_pri_sig_v2 = SHA-256(signingPayload)
+```
+
+Where `signingPayload` is exactly:
+
+```text
+FRESHCONTEXT_HA_PRI_V2
+result_id=<resultId>
+canonical_content_sha256=<canonicalContentSha256>
+semantic_fingerprint_sha256=<semanticFingerprintSha256>
+adapter=<adapter>
+published_at=<publishedAt-or-null>
+retrieved_at=<retrievedAt-or-null>
+engine_version=<engineVersion>
+```
+
+### Field Meaning
+
+- `FRESHCONTEXT_HA_PRI_V2`: schema/version string
+- `result_id`: stable row identifier
+- `canonical_content_sha256`: cryptographic digest of canonical raw content
+- `semantic_fingerprint_sha256`: cryptographic digest of semantic identity fields
+- `adapter`: source adapter name
+- `published_at`: source/content publication timestamp, or explicit null sentinel
+- `retrieved_at`: retrieval or collection timestamp, or explicit null sentinel
+- `engine_version`: scoring/signature engine version
+
+Store/Ledger systems may map `scraped_at` to the v2 `retrieved_at` signing field.
+
+## Verification Model
+
+Future read or audit verification should:
+
+1. Load the stored row.
+2. Recompute canonical raw content from stored content fields.
+3. Recompute `canonical_content_sha256`.
+4. Recompute semantic identity fields.
+5. Recompute `semantic_fingerprint_sha256`.
+6. Recompute `ha_pri_sig_v2` from the canonical field sequence.
+7. Compare stored vs recomputed values.
+8. Mark the result:
+   - `valid`
+   - `invalid`
+   - `unknown`
+9. Surface verification status to internal/debug paths first.
+10. Avoid silently trusting unverifiable rows.
+
+Verification must not mutate old rows during read unless a dedicated migration explicitly allows it.
+
+## Backward Compatibility
+
+Ha-Pri v2 should not remove or reinterpret v1.
+
+Rules:
+
+- Keep `ha_pri_sig` readable.
+- Add `ha_pri_sig_v2` separately.
+- Treat old rows without v2 fields as `unknown`, not invalid.
+- Do not reject old rows solely because they lack v2.
+- Preserve v1 formula tests.
+- Add v2 fixtures before any production write path changes.
+
+## Future HMAC Boundary
+
+HMAC-SHA256 may be useful later if FreshContext needs origin authentication rather than only tamper evidence.
+
+That would require:
+
+- a private deployment key
+- secret rotation
+- key identifiers
+- verification policy for old key versions
+- clear trust-boundary documentation
+
+This phase does not add HMAC, secrets, or key management.
+
+## Suggested Future Patch Sequence
+
+1. Add pure canonicalization helpers and deterministic tests.
+2. Add pure v2 signature helper and fixtures.
+3. Add optional verification helper that returns `valid`, `invalid`, or `unknown`.
+4. Add D1 columns in a separate schema phase.
+5. Write v2 fields for new rows only.
+6. Expose verification status on debug/internal endpoints first.
+7. Decide later whether public feed output should include v2 status.
+
+## Non-Goals
+
+This design does not:
+
+- change runtime behavior
+- change scoring
+- change MCP tool schemas
+- change D1 schema
+- change Worker write paths
+- migrate old rows
+- add HMAC
+- add secrets
+- reject rows in production
+- publish npm
+- deploy the Worker