freshcontext-mcp 0.3.18 → 0.3.20

package/FRESHCONTEXT_SPEC.md

@@ -0,0 +1,317 @@
+# The FreshContext Specification
+**Version 1.2 — May 2026**
+*Authored by Immanuel Gabriel (Prince Gabriel) — Grootfontein, Namibia*
+
+---
+
+## What This Is
+
+The FreshContext Specification defines a standard envelope format for AI-retrieved web data and a temporal correction layer that scores it.
+
+It exists to solve one problem: **AI models present stale data with the same confidence as fresh data, and users have no way to tell the difference.**
+
+> **Illustrative demonstration:** [freshcontext-mcp.gimmanuel73.workers.dev/demo](https://freshcontext-mcp.gimmanuel73.workers.dev/demo) — same model, same query, different answers from different ranked context. The demo shows how FreshContext treats freshness signals.
+
+FreshContext addresses this by wrapping retrieved content in a structured envelope that carries three explicit properties:
+
+1. **When** the data was retrieved (exact ISO 8601 timestamp)
+2. **Where** it came from (canonical source URL)
+3. **How confident** we are that the content date is accurate (freshness confidence)
+
+Any tool, agent, or system that implements this spec is **FreshContext-compatible**.
+
+The specification is one half of the system. The other half — the Decay-Adjusted Relevancy (DAR) engine that scores wrapped signals — is documented separately in [METHODOLOGY.md](./METHODOLOGY.md).
+
+---
+
+## The Envelope Format
+
+Every FreshContext-compatible response MUST wrap its content in the following envelope:
+
+```
+[FRESHCONTEXT]
+Source: <canonical_url>
+Published: <content_date_or_"unknown">
+Retrieved: <iso8601_timestamp>
+Confidence: <high|medium|low>
+---
+<content>
+[/FRESHCONTEXT]
+```
+
+### Field Definitions
+
+| Field | Required | Format | Description |
+|---|---|---|---|
+| `Source` | Yes | Valid URL | The canonical URL of the original source |
+| `Published` | Yes | ISO 8601 date or `"unknown"` | Best estimate of when the content was originally published |
+| `Retrieved` | Yes | ISO 8601 datetime with timezone | Exact timestamp when this data was fetched |
+| `Confidence` | Yes | `high`, `medium`, or `low` | Confidence level of the `Published` date estimate |
+
+---
+
+## Confidence Levels
+
+### `high`
+The publication date was sourced from a structured, machine-readable field — an API response, HTML metadata tag, RSS feed, or official timestamp. The date is reliable.
+
+*Examples: GitHub API `pushed_at`, arXiv submission date, Hacker News `created_at`, SEC EDGAR filing date, USASpending.gov award date*
+
+### `medium`
+The publication date was inferred from page signals — visible date strings, URL patterns, or content heuristics. Likely correct but not guaranteed.
+
+*Examples: Blog post date parsed from HTML, URL containing `/2025/03/`, footer copyright year*
+
+### `low`
+No reliable date signal was found. The date is an estimate based on indirect signals or is entirely unknown.
+
+*Examples: Static page with no date, scraped content with no metadata, cached result of unknown age*
+
+---
+
+## Structured Form (JSON)
+
+Implementations MAY additionally expose freshness metadata as structured JSON alongside the text envelope:
+
+```json
+{
+  "freshcontext": {
+    "source_url": "https://github.com/owner/repo",
+    "content_date": "2026-03-05",
+    "retrieved_at": "2026-03-16T09:19:00.000Z",
+    "freshness_confidence": "high",
+    "adapter": "github",
+    "freshness_score": 94
+  },
+  "content": "..."
+}
+```
+
+### `freshness_score` (optional)
+
+A numeric representation of data freshness from 0-100. The canonical FreshContext model is exponential Decay-Adjusted Relevancy (DAR):
+
+```
+R_t = R_0 · e^(-λt)
+```
+
+Where:
+- `R_0` is the starting relevance or freshness value.
+- `λ` is the source-specific decay constant.
+- `t` is the age of the signal, normally measured from `content_date` / `published_at` to retrieval or query time.
+- `R_t` is the decay-adjusted relevance or freshness value.
+
+For simple envelope-level `freshness_score`, `R_0` MAY be normalised to `100`. For ranked intelligence feeds, `R_0` MAY be semantic base relevance, profile relevance, or adapter-specific utility. The displayed `freshness_score` remains normalised to 0-100 for compatibility.
+
+FreshContext scoring MAY be context-conditioned. Implementations that rank multiple signals MAY compute `R_0` from the requesting user, query, agent, workflow, semantic similarity, profile relevance, or adapter-specific relevance before applying DAR. The envelope contract remains the same: systems SHOULD expose source, publication time, retrieval time, confidence, and freshness metadata regardless of how `R_0` is computed.
+
+Implementations SHOULD use source-specific `λ` values to reflect how quickly different categories of data lose temporal utility.
+
+#### Reference Decay Classes
+
+These are reference/default classes. Implementations MAY tune them by source, user profile, or deployment while preserving the exponential DAR model.
+
+| Source class | Example λ (per hour) | Approximate half-life | Examples |
+|---|---|---|---|
+| Fast discussion | 0.050 | ~14 hours | Hacker News front-page stories |
+| News cycle | 0.020 | ~35 hours | GDELT global news |
+| Community / launch signals | 0.010 | ~3 days | Reddit, Product Hunt |
+| Jobs / material events | 0.005 | ~6 days | Job listings, SEC 8-K filings |
+| Procurement / market context | 0.001 | ~29 days | Stooq quotes, gov contracts, YC company data |
+| Package / changelog cadence | 0.0005 | ~58 days | npm/PyPI, GitHub Releases |
+| Repository activity | 0.0002 | ~5 months | GitHub repositories |
+| Academic work | 0.00005 | ~1.6 years | arXiv, Google Scholar |
+
+#### Score Interpretation
+
+| Score | Interpretation |
+|---|---|
+| 90–100 | Very recent for its source class — treat as current |
+| 70–89 | Still fresh enough for most uses |
+| 50–69 | Decayed but potentially useful — verify before acting |
+| Below 50 | Low temporal utility — use with caution |
+
+### Missing, Invalid, and Future Timestamps
+
+FreshContext-compatible systems MUST NOT treat missing or invalid timestamps as fresh.
+
+- If no reliable timestamp exists, implementations SHOULD set `freshness_score` to `null` and `freshness_confidence` to `low`.
+- Missing timestamps MUST NOT be treated as current.
+- Invalid timestamps MUST NOT be treated as current.
+- Future timestamps beyond a small clock-skew tolerance MUST NOT be treated as current.
+- Implementations SHOULD surface timestamp uncertainty explicitly.
+
+---
+
+## Adapter Contract
+
+Any data source that feeds into a FreshContext-compatible system is called an **adapter**. Adapters MUST:
+
+1. Return raw content plus a `content_date` (or `null` if unknown)
+2. Set a `freshness_confidence` level based on how the date was determined
+3. Never fabricate or forward-date content timestamps
+4. Clearly identify which source system produced the data via the `adapter` field
+5. Surface retrieval failures explicitly
+
+Adapters SHOULD:
+
+- Prefer structured API sources over scraped content when both are available
+- Log retrieval errors without silently returning cached or stale data
+- Surface rate-limit or access-denied errors explicitly rather than returning empty content
+- Use source-specific decay constants from the reference decay classes above
+
+### Failure Honesty
+
+Adapters MUST NOT wrap failed, empty, blocked, timeout, rate-limited, access-denied, or malformed output as high-confidence fresh context.
+
+If adapter output looks like an error, implementations SHOULD:
+
+- Downgrade `freshness_confidence` to `low`
+- Set `freshness_score` to `null`
+- Preserve enough error context for diagnosis
+- Avoid presenting the result as successful current context
+
+---
+
+## Composite Adapters
+
+A **composite adapter** is a FreshContext-compatible adapter that calls multiple upstream adapters in parallel and combines their results into a single unified response. Each upstream result MUST retain its own FreshContext envelope — the composite wrapper MUST NOT collapse individual timestamps into a single envelope.
+
+Composite adapters SHOULD:
+
+- Fire all upstream calls in parallel (e.g. `Promise.allSettled`)
+- Handle partial failures gracefully — if one upstream fails, return the rest
+- Label each section clearly with its source adapter name
+- Include a composite `retrieved_at` timestamp representing the time the composite call was initiated
+
+*Examples in the reference implementation: `extract_landscape`, `extract_company_landscape`, `extract_idea_landscape`*
+
+---
+
+## Why This Matters for AI Agents
+
+Large language models have no internal clock. When an agent retrieves web data, it cannot distinguish between something published this morning and something published three years ago — unless that information is explicitly surfaced.
+
+Without FreshContext (or equivalent):
+- An agent recommending job listings may recommend roles that no longer exist
+- An agent summarising market trends may cite conditions from a previous cycle
+- An agent checking a competitor's pricing may act on outdated information
+- An agent synthesising news may present last year's controversy as current
+
+With FreshContext:
+- Every piece of retrieved data carries its own timestamp
+- The agent can reason about data age before acting
+- Users can see exactly how fresh their AI's information is
+- Composite intelligence reports carry per-source freshness signals
+
+---
+
+## Compatibility
+
+A tool, server, or API is **FreshContext-compatible** if:
+
+- Its responses include the `[FRESHCONTEXT]...[/FRESHCONTEXT]` envelope, OR
+- Its responses include the structured JSON form with `freshcontext.retrieved_at` and `freshcontext.freshness_confidence` fields
+
+Partial implementations that include only `retrieved_at` without `freshness_confidence` are considered **FreshContext-aware** but not fully compatible.
+
+### Compatibility Levels
+
+| Level | Requirements |
+|---|---|
+| **FreshContext-compatible** | Full envelope OR full JSON form with `retrieved_at` + `freshness_confidence` |
+| **FreshContext-aware** | Includes `retrieved_at` but not `freshness_confidence` |
+| **FreshContext-scored** | Full compatible + numeric `freshness_score` with domain-specific decay |
+
+MCP is one interface, not the whole FreshContext model. The envelope and structured JSON metadata are the compatibility contract. FreshContext-compatible systems may be MCP servers, APIs, CLIs, npm packages, dashboards, agents, or internal services.
+
+---
+
+## Core-backed Implementation Status
+
+FreshContext Core now owns the envelope/scoring layer in the reference implementation:
+
+- Freshness scoring
+- Envelope generation
+- Confidence metadata
+- Failure guards
+- Shared types
+- Rank/explain primitives
+
+The live MCP Worker uses Core-backed envelope generation while Cloudflare-specific runtime behavior, MCP transport, cache policy, rate limits, and deployment concerns remain outside Core.
+
+Claude Desktop is a supported MCP client, but FreshContext Core is not Claude-dependent. MCP is one interface over the methodology; FreshContext-compatible systems may also expose results through npm packages, APIs, CLIs, dashboards, or other agent runtimes.
+
+---
+
+## Reference Implementation
+
+The canonical reference implementation of this specification is:
+
+**freshcontext-mcp@0.3.19** — an MCP server with `evaluate_context` for caller-provided candidate context plus 21 read-only reference adapters covering:
+
+**Intelligence:** GitHub, Hacker News, Google Scholar, arXiv, Reddit
+
+**Competitive research:** YC Companies, Product Hunt, GitHub repo search, npm/PyPI package trends
+
+**Market data:** Stooq quote data (up to 5 tickers), job listings (Remotive, RemoteOK, HN Hiring)
+
+**Official, regulatory, and procurement sources:**
+- `extract_changelog` — release history from any repo, npm package, or website
+- `extract_govcontracts` — US federal contract awards (USASpending.gov)
+- `extract_sec_filings` — SEC 8-K material event disclosures (EDGAR)
+- `extract_gdelt` — global news intelligence, 100+ languages, updated every 15 minutes
+- `extract_gebiz` — Singapore Government procurement (data.gov.sg)
+
+**Composite landscapes:** `extract_landscape`, `extract_idea_landscape`, `extract_gov_landscape`, `extract_finance_landscape`, `extract_company_landscape`
+
+**Deployment:**
+- npm: `freshcontext-mcp`
+- GitHub: https://github.com/PrinceGabriel-lgtm/freshcontext-mcp
+- Cloud endpoint: `https://freshcontext-mcp.gimmanuel73.workers.dev/mcp`
+- MCP Registry: `io.github.PrinceGabriel-lgtm/freshcontext`
+
+---
+
+## Changelog
+
+### Version 1.2 — May 2026
+- Clarified exponential DAR scoring as the canonical freshness model.
+- Added missing/invalid/future timestamp handling guidance.
+- Added failure-honesty requirements for adapter output.
+- Added Core-backed envelope/scoring implementation status.
+- Added optional context-conditioned scoring language without changing the envelope contract.
+- Clarified MCP as one interface over the FreshContext methodology.
+- Updated reference implementation language for freshcontext-mcp@0.3.19, `evaluate_context`, and 21 read-only reference adapters.
+
+### Version 1.1 — April 2026
+- Added Composite Adapters section
+- Added domain-specific decay rate table with recommended values
+- Added Compatibility Levels table (compatible / aware / scored)
+- Updated reference implementation to 21 reference adapter tools
+- Added `extract_gdelt`, `extract_gebiz`, `extract_sec_filings` to high-confidence examples
+- Added Apify Store and MCP Registry to reference implementation listings
+
+### Version 1.0 — March 2026
+- Initial specification published
+
+---
+
+## Versioning
+
+This document is version 1.2 of the FreshContext Specification.
+
+Future versions will be tagged in this repository. Breaking changes to the envelope format will increment the major version. Additive changes (new optional fields, new confidence levels, new recommended values) will increment the minor version.
+
+---
+
+## License
+
+This specification is published under the MIT License.
+Implementations may be proprietary or open source.
+Attribution to the FreshContext Specification is appreciated but not required.
+
+---
+
+*"The work isn't gone. It's just waiting to be continued."*
+*— Prince Gabriel, Grootfontein, Namibia*