freshcontext-mcp 0.3.17 → 0.3.19

package/docs/CORE_API.md

@@ -0,0 +1,226 @@
+# FreshContext Core API
+
+FreshContext Core is the reusable engine layer in the current integrated Core/MCP package. It owns signal normalization, envelope creation, freshness scoring, failure honesty, Source Profiles, decision output, rank/explain primitives, the context-utility primitive, and pure provenance helpers.
+
+MCP, Worker HTTP, future REST, and future CLI/SDK surfaces should use Core as the contract center instead of redefining freshness or envelope behavior per host.
+
+For the package-level boundary between Core, MCP, adapters, and deployment surfaces, see [Core / MCP Boundary](./CORE_MCP_BOUNDARY.md).
+
+## Stable Public Core API
+
+Import stable Core functions from:
+
+```ts
+import {
+  calculateFreshnessScore,
+  formatForLLM,
+  looksLikeFailedAdapterContent,
+  scoreLabel,
+  stampFreshness,
+  toStructuredJSON,
+} from "./src/core/index.js";
+```
+
+### Envelope
+
+- `stampFreshness(result, options, adapter)` creates a `FreshContext` object from adapter output.
+- `formatForLLM(ctx, options?)` renders the text envelope and trailing structured JSON block.
+- `toStructuredJSON(ctx)` returns the machine-readable FreshContext JSON shape.
+
+### Scoring
+
+- `calculateFreshnessScore(content_date, retrieved_at, adapter)` returns a freshness score from `0..100`, or `null` when the content date cannot be trusted.
+- `scoreLabel(score)` maps a numeric freshness score to a human-readable label.
+
+### Guards
+
+- `looksLikeFailedAdapterContent(raw)` detects empty, security, timeout, and error-like adapter output so failed content is not stamped as fresh high-confidence context.
+
+## Core Evaluation Pipeline
+
+The Core evaluation pipeline is the pure orchestration layer over the existing primitives.
+
+Public exports:
+
+- `evaluateSignal(input, options?)`
+- `evaluateSignals(inputs, options?)`
+- `CoreSignalEvaluationOptions`
+- `CoreSignalEvaluationResult`
+- `CoreSignalEnvelopeResult`
+- `CoreSignalProvenanceOptions`
+
+`evaluateSignal` normalizes a signal, applies timestamp/failure guards, computes `freshness_score`, computes context-conditioned utility, ranks/explains the signal, optionally creates a FreshContext envelope, and optionally prepares Ha-Pri v2 provenance material.
+
+It does not fetch, cache, write D1, inspect Worker bindings, know MCP tool schemas, deploy, or publish. Hosts decide whether to store, cache, transmit, or expose the returned result.
+
+`evaluateSignals` evaluates each input and returns evaluations sorted by existing `rankSignal` final score, preserving input order when scores tie. Context utility is returned as a sidecar and does not replace `final_score`.
+
+Context utility is returned as sidecar output in the current pipeline; it does not replace or modify the default `rankSignal` / `evaluateSignals` ordering. A future pass may add an explicit utility-weighted ranking mode.
+
+Local demo:
+
+```bash
+npm run demo:evaluate:file -- examples/sources.academic.example.json
+npm run demo:evaluate:file -- examples/sources.jobs.example.json
+```
+
+The demo reads caller-provided JSON with `profile`, `intent`, and `signals`, then returns decision-first output. It does not fetch URLs, crawl, read folders, deploy REST, or implement Operator mode.
+
+### Stable Types
+
+- `FreshContext`
+- `AdapterResult`
+- `ExtractOptions`
+- `EnvelopeFormatOptions`
+- `SignalConfidence`
+
+These types describe the stable envelope and adapter result contract.
+
+## Signal Contract v1
+
+Signal Contract v1 is the additive Core shape for a retrieved signal before it is ranked, wrapped, stored, or passed to an agent workflow.
+
+Public exports:
+
+- `SIGNAL_CONTRACT_VERSION`
+- `normalizeSignal(input, options?)`
+- `FreshContextSignalInput`
+- `FreshContextSignal`
+- `SignalDateConfidence`
+- `SignalContractVersion`
+- `SignalNormalizeOptions`
+
+`published_at` is the canonical signal timestamp. `content_date` is accepted as an adapter/envelope compatibility alias. Normalization clears invalid or meaningfully future-dated timestamps, marks failed/error-looking content as `status: "failed"`, clamps `semantic_score` into `0..1`, and records normalization reasons.
+
+See [Signal Contract v1](./SIGNAL_CONTRACT.md).
+
+## Source Profiles
+
+Source Profiles are early public Core metadata for describing how classes of information age, fail, rank, and explain.
+
+Public exports:
+
+- `BUILT_IN_SOURCE_PROFILES`
+- `getSourceProfile(profileId)`
+- `listSourceProfiles()`
+- `SourceProfile`
+- `SourceProfileId`
+- `SourceAuthorityHint`
+- `SourceDatePolicy`
+- `SourceFailurePolicy`
+- `SourceSurface`
+
+They reframe the 21 named adapter tools as reference adapters and source-profile examples instead of the product identity. The MCP server also exposes `evaluate_context` as the generic caller-provided context evaluation path. Source Profiles do not implement `retrieve(...)`, Operator mode, adapter selection, crawling, local file search, or any host/runtime behavior.
+
+## Decision Helper
+
+The decision helper translates a Core evaluation result into user-facing action meaning.
+
+Public exports:
+
+- `interpretEvaluation(evaluation, options?)`
+- `interpretEvaluations(evaluations, options?)`
+- `ContextDecision`
+- `IntentProfileId`
+- `ContextDecisionOptions`
+- `ContextDecisionResult`
+
+Supported decisions:
+
+- `use_first`
+- `cite_as_primary`
+- `cite_as_supporting`
+- `use_as_background`
+- `needs_verification`
+- `needs_refresh`
+- `watch_only`
+- `exclude`
+
+Supported intent profiles:
+
+- `citation_check`
+- `student_research`
+- `developer_adoption`
+- `job_search`
+- `market_watch`
+- `business_due_diligence`
+- `medical_literature_triage`
+
+The helper consumes existing `CoreSignalEvaluationResult` fields plus optional Source Profile metadata. It does not change `evaluateSignal`, `evaluateSignals`, `rankSignal`, ranking order, freshness math, utility math, envelopes, provenance, or host behavior.
+
+FreshContext decisions judge citation readiness, context usefulness, freshness, traceability, and uncertainty. They do not certify truth or provide legal, medical, tax, employment, academic, or investment advice.
+
+Demo output will be updated separately so presentation stays separate from Core decision logic.
+
+## Public Ranking Primitives
+
+The ranking primitives are public, but consumers should treat their score scales carefully:
+
+- `rankSignal(signal, options?)`
+- `rankSignals(signals, options?)`
+- `explainSignal(rankedSignalLike)`
+- `FreshSignal`
+- `RankedSignal`
+- `RankOptions`
+
+Score scales:
+
+- `semantic_score`: normalized `0..1`
+- `final_score`: normalized `0..1`
+- `freshness_score`: FreshContext freshness score `0..100`, or `null`
+
+Ranking combines semantic relevance and freshness into a deterministic order. It does not own retrieval, embedding, vector search, storage, or host-specific scoring policy.
+
+## Experimental Utility Primitive
+
+The context-conditioned utility primitive is pure and tested, but it is not production-wired into MCP ranking, Worker feeds, Store scoring, or runtime behavior.
+
+Experimental exports:
+
+- `calculateContextUtility`
+- `ContextUtilityStatus`
+- `ContextUtilityInput`
+- `ContextUtilityResult`
+
+These are pure Core math. They are now connected inside `evaluateSignal` as sidecar utility output, but they are not production-wired into MCP ranking, Worker feeds, Store scoring, or runtime behavior.
+
+## Provenance Helpers
+
+Ha-Pri v2 is available as pure Core helper functionality:
+
+- `canonicalizeHaPriContent`
+- `sha256Hex`
+- `calculateHaPriV2`
+- `verifyHaPriV2`
+- `HaPriV2Input`
+- `HaPriV2Result`
+- `HaPriV2VerificationResult`
+
+`evaluateSignal` can optionally prepare Ha-Pri v2 material when `includeProvenance` is set and required input material is present. Core does not persist provenance, add D1 columns, verify rows on read, reject rows, or replace Worker Ha-Pri v1 behavior.
+
+## Internal, Policy, and Compatibility Exports
+
+- `clampScore` is an internal ranking helper. It is currently exported for tests and utility use, but it should not be presented as a primary buyer-facing API.
+- `LAMBDA` is the current policy constant table used by freshness scoring. It documents the reference decay policy, but it is not a buyer-facing tuning API.
+
+Compatibility lanes should remain:
+
+- `src/types.ts` re-exports legacy adapter types from Core.
+- `src/tools/freshnessStamp.ts` re-exports envelope helpers for older MCP/npm import paths.
+
+These lanes protect existing imports while Core becomes the center. Do not remove them until downstream imports have been migrated intentionally.
+
+## What Core Does Not Own
+
+Core does not own:
+
+- MCP transport
+- Cloudflare runtime behavior
+- KV cache policy
+- Cache metadata injection
+- D1, feed, or cron behavior
+- Store/feed scoring and provenance persistence
+- Hosted dashboard, API, deployment, or runtime concerns
+
+Hosts may wrap Core outputs with their own transport, cache, session, rate-limit, or persistence metadata, but they should not fork the Core envelope and freshness contract without an explicit compatibility reason.
+

package/docs/CORE_MCP_BOUNDARY.md

@@ -0,0 +1,106 @@
+# FreshContext Core / MCP Boundary
+
+FreshContext is the context judgment layer between retrieval and reasoning.
+
+The current npm package is intentionally named `freshcontext-mcp` because MCP is the first live interface. That does not make MCP the whole product. MCP is a host layer over the FreshContext Core engine.
+
+## Product Shape
+
+```text
+Candidate context
+  -> FreshContext Core
+  -> decision-ready context
+  -> MCP / Worker / future REST / future SDK / future CLI
+```
+
+FreshContext Core owns the judgment contract:
+
+- signal normalization
+- freshness scoring
+- source profiles
+- context utility sidecar scoring
+- rank and explanation primitives
+- decision helper output
+- envelope and provenance helpers
+
+MCP owns the live reference interface:
+
+- tool registration
+- MCP input schemas
+- stdio transport
+- client-facing tool descriptions
+- formatting Core/adapter output for MCP clients
+
+Adapters own source intake:
+
+- source-specific fetching
+- source-specific parsing
+- timestamp extraction
+- source-specific normalization
+- failure normalization before Core evaluation
+
+Worker/site surfaces own deployment concerns:
+
+- Cloudflare runtime behavior
+- KV/D1/cache/feed concerns
+- hosted demo/site presentation
+- runtime guards and deployment configuration
+
+## Current Live Boundary
+
+Live today:
+
+- npm package: `freshcontext-mcp@0.3.19`
+- MCP stdio server and published binary: `freshcontext-mcp`
+- `evaluate_context` MCP tool for caller-provided candidate context
+- 21 named read-only reference adapters
+- Core signal evaluation
+- Source Profiles
+- Decision Helper
+- adapter registry metadata
+- arXiv signal-to-decision proof
+- bring-your-own-context local demos
+- Trust Scanner release gate
+
+Not live today:
+
+- standalone Core npm package
+- package rename to `freshcontext`
+- Operator / `retrieve(...)`
+- automatic browser crawling
+- automatic local folder/PDF scanning
+- hosted enterprise dashboard or billing
+- hard Ha-Pri v2 production enforcement
+- full adapter ingestion into pure signal paths
+
+## Future Package Split
+
+The safe split path is staged:
+
+1. Keep `freshcontext-mcp` stable for current users.
+2. Maintain Core as a pure internal export surface.
+3. Audit Core dependencies, Node/browser compatibility, and API stability.
+4. Publish a standalone Core package only after compatibility tests exist.
+5. Make `freshcontext-mcp` depend on the standalone Core package.
+6. Consider a repo rename to `freshcontext` only after package/client links are stable.
+
+The expected future shape is:
+
+```text
+freshcontext
+  packages/core      reusable judgment engine
+  packages/adapters  reference source intake assets
+  packages/mcp       MCP host layer
+  packages/cli       future local evaluator
+  docs
+```
+
+## Compatibility Rule
+
+Do not remove the current compatibility lanes until a dedicated migration pass exists:
+
+- `src/types.ts` re-exports legacy adapter types from Core.
+- `src/tools/freshnessStamp.ts` re-exports envelope helpers for older MCP/npm import paths.
+- `dist/server.js` remains the package `main` and MCP binary target.
+
+The architecture direction is clear, but the public runtime should stay boring and stable.

package/docs/DEPENDENCY_DILIGENCE.md

@@ -0,0 +1,63 @@
+# FreshContext Dependency Diligence Notes
+
+This document records dependency and license diligence notes from the Trust L4/L5 cleanup. It is not legal advice and does not replace professional review for external review, distribution, or formal diligence.
+
+## Current Audit Status
+
+As of Pass 8-AB:
+
+- `npm audit --omit=dev`: clean.
+- `npm audit`: clean.
+- The published MCP npm package excludes the Apify Actor entrypoint and does not install Apify/Crawlee in normal consumer installs.
+- The previous moderate `qs` and `ws` advisories were resolved with narrow transitive overrides in the source checkout.
+- No package version change was made.
+
+## Resolved Advisories
+
+`qs`
+
+- Previous severity: moderate.
+- Path: `@modelcontextprotocol/sdk -> express/body-parser -> qs`.
+- Resolution: pinned through npm `overrides` to `qs@6.15.2`.
+
+`ws`
+
+- Previous severity: moderate.
+- Historical source-checkout path: `apify -> ws`.
+- Resolution: pinned through npm `overrides` to `ws@8.20.1` for source-checkout Apify Actor workflows.
+
+`file-type`
+
+- Previous severity: moderate in fresh consumer installs.
+- Historical consumer path: `apify -> @crawlee/utils -> file-type`.
+- Resolution: Apify/Crawlee were removed from the normal published MCP package dependency surface. Apify remains a source-checkout / separate-actor concern.
+
+## License Inventory Notes
+
+The Trust L4 license inventory was broadly permissive, including MIT, Apache-2.0, BSD variants, ISC, 0BSD, BlueOak-1.0.0, and similar permissive variants.
+
+No GPL, AGPL, LGPL, MPL, EPL, CDDL, or similar copyleft licenses were reported in the Trust L4 scan.
+
+`map-stream@0.1.0`
+
+- Scanner result: `UNKNOWN`.
+- Path observed during L4: transitive through source-checkout `apify` / Crawlee-related dependencies.
+- Diligence note: package metadata appears incomplete, but the installed package includes an MIT-style license file.
+- Action: keep as a source-checkout / actor-packaging diligence note and recheck before external diligence or Apify Actor distribution.
+
+`caniuse-lite`
+
+- Scanner result: `CC-BY-4.0`.
+- Diligence note: preserve and review attribution requirements before external diligence, bundled distribution, or a formal review package.
+
+## Before External Diligence or Distribution
+
+- Rerun `npm audit --omit=dev`.
+- Rerun `npm audit`.
+- Rerun dependency license inventory.
+- Review scanner-unknown packages.
+- Review `caniuse-lite` attribution requirements.
+- Generate an SBOM if requested by an evaluator, reviewer, or downstream distributor.
+- Review dependency and license posture with qualified counsel when external distribution or formal diligence requires it.
+
+Do not treat this document as a legal conclusion.

package/docs/FUTURE_LANES.md

@@ -0,0 +1,173 @@
+# FreshContext Future Lanes
+
+This document keeps future FreshContext work organized without turning roadmap ideas into public claims.
+
+FreshContext is live today as an integrated MCP/Core package. Future work should stay in lanes, start with audits, and avoid feature sprawl.
+
+The current package boundary is documented in [Core / MCP Boundary](./CORE_MCP_BOUNDARY.md). Treat MCP as the first live host over FreshContext Core, not as the whole product identity.
+
+## Current Live Boundary
+
+Live today:
+
+- npm package: `freshcontext-mcp@0.3.19`
+- MCP stdio server
+- `evaluate_context` MCP tool for caller-provided candidate context
+- 21 read-only reference adapters
+- Core signal evaluation
+- Source Profiles
+- Decision Helper
+- adapter registry metadata
+- arXiv signal-to-decision proof
+- bring-your-own-context local demos
+- Trust Scanner release gate
+
+Not live today:
+
+- Operator / `retrieve(...)`
+- browser crawling
+- automatic local file, folder, or PDF scanning
+- hosted dashboard or billing
+- hard Ha-Pri v2 production enforcement
+- standalone Core SDK package
+- full adapter ingestion
+
+## Lane 1: Client Setup Reliability
+
+Goal:
+
+```text
+Make Claude, Codex, and MCP-compatible clients connect reliably to the published package.
+```
+
+Start with setup guide audits, Claude Desktop local/global package paths, Codex local MCP config paths, stale global package fixes, and smoke command expectations.
+
+Do not claim ChatGPT/OpenAI connector compatibility until a separate compatibility audit is done.
+
+## Lane 2: Generic Context Evaluation
+
+Goal:
+
+```text
+Let any caller provide candidate context and get decision-ready output.
+```
+
+Current live MCP path:
+
+```text
+evaluate_context
+```
+
+Hard boundary:
+
+```text
+No fetching, crawling, browsing, folder reading, or retrieval orchestration.
+Only evaluate caller-provided candidate context.
+```
+
+Next work in this lane should focus on CLI, SDK, and REST ergonomics over the same caller-provided signal shape.
+
+## Lane 3: Multi-Agent Context Handoff Proof
+
+Goal:
+
+```text
+Show FreshContext as an independent context judgment layer between agents.
+```
+
+Proof shape:
+
+```text
+agent A produces candidate context
+-> FreshContext evaluates it
+-> agent B receives decision-ready context
+```
+
+Do not build a full multi-agent framework. Prove the handoff boundary.
+
+## Lane 4: Core SDK Extraction Audit
+
+Goal:
+
+```text
+Decide whether Core should become a standalone package.
+```
+
+Audit current Core exports, dependency boundaries, package shape, browser/node compatibility, public API stability, and what remains MCP-only.
+
+No extraction without a compatibility plan. Keep `freshcontext-mcp` stable until a standalone Core package has compatibility tests and a migration path.
+
+## Lane 5: Local/User Data Intake Audit
+
+Goal:
+
+```text
+Explore student, research, and local-PC workflows safely.
+```
+
+Candidate sources include notes, PDFs, local JSON/CSV, citation exports, and database rows.
+
+Hard boundary:
+
+```text
+Consent-first design. No automatic folder scanning or background file reading.
+```
+
+## Lane 6: Decision Layer Upgrade
+
+Goal:
+
+```text
+Make decisions more useful without silently changing ranking.
+```
+
+Possible inputs include context utility, control signal, future context signal, confidence tiers, and source-profile-specific thresholds.
+
+Do not make `utility.score` affect ranking by default without a dedicated ranking policy pass.
+
+## Lane 7: Ha-Pri v2 Production Path
+
+Goal:
+
+```text
+Turn Ha-Pri v2 from pure Core helper/design into production enforcement where appropriate.
+```
+
+Audit canonical content material, storage path, verification timing, failure mode, D1/Worker compatibility, and migration plan.
+
+Do not claim hard tamper enforcement until the read/write path exists.
+
+## Lane 8: GDELT GKG / Richer Source Intelligence
+
+Goal:
+
+```text
+Upgrade GDELT intelligence with richer global knowledge graph signals.
+```
+
+Possible fields include tone, Goldstein scale, event codes, actor geography, theme density, and source timeline.
+
+Do not mix this with generic context evaluation or local intake.
+
+## Lane 9: Operator / Retrieve Orchestration
+
+Goal:
+
+```text
+Coordinate retrieval only after the decision layer and adapter boundaries are mature.
+```
+
+Operator may later select adapters, call retrievers, refresh stale sources, and package best context.
+
+Not now. Operator is a later workflow layer over Core and adapters.
+
+## Operating Rule
+
+Every lane starts with:
+
+```text
+audit -> small patch -> validation -> stop
+```
+
+Do not combine lanes because the architecture is tempting.
+