freshcontext-mcp 0.3.10 → 0.3.12

package/.actor/Dockerfile

@@ -0,0 +1,16 @@
+FROM apify/actor-node:20
+
+# Copy package files first for better Docker layer caching
+COPY package*.json ./
+
+# Install all dependencies including apify SDK
+RUN npm install --include=dev
+
+# Copy source and pre-built dist
+COPY . ./
+
+# Rebuild TypeScript (dist/ from repo is fallback if this fails)
+RUN npm run build || echo "Build had warnings, using pre-compiled dist/"
+
+# Tell Apify to run the Actor entry point, not the MCP server
+CMD ["node", "dist/apify.js"]

package/.actor/actor.json

@@ -4,5 +4,6 @@
   "title": "FreshContext MCP",
   "version": "0.3.1",
   "input": "../input_schema.json",
-  "output": "./output_schema.json"
+  "output": "./output_schema.json",
+  "dockerfile": "./Dockerfile"
 }

package/.github/workflows/publish.yml

@@ -0,0 +1,32 @@
+name: Build and Publish
+
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  build-and-publish:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Node.js 18
+        uses: actions/setup-node@v4
+        with:
+          node-version: '18'
+          registry-url: 'https://registry.npmjs.org'
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Build
+        run: npm run build
+
+      - name: Publish to npm
+        run: npm publish --access public
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
+        continue-on-error: true

package/ARCHITECTURE_UPGRADE_CHECKLIST.md

@@ -0,0 +1,88 @@
+# FreshContext — Architecture Upgrade Checklist
+**Date started:** 2026-03-19
+**Author:** Prince Gabriel, Grootfontein, Namibia
+
+---
+
+## [ ] Upgrade 1 — freshness_score numeric field
+Implement the 0-100 numeric score defined in FRESHCONTEXT_SPEC.md.
+Formula: max(0, 100 - (days_since_retrieved * decay_rate))
+Location: src/tools/freshnessStamp.ts
+Decay rates by adapter: finance=5.0, jobs=3.0, hackernews=2.0, github=1.0, scholar=0.3, default=1.5
+Adds the score to both the text envelope and the JSON form.
+Makes FreshContext fully spec-compliant by your own standard.
+Cost: zero.
+
+---
+
+## [x] Upgrade 2 — Cloudflare KV response caching  ← DONE (already implemented in worker.ts)
+Cache adapter results in KV with adapter-specific TTLs so the same query
+hitting the Worker twice doesn't make two upstream API calls.
+Cache key: sha256(tool + ":" + url)
+TTLs: HN/Reddit = 1 hour, GitHub/YC = 6 hours, govcontracts/scholar = 24 hours
+Location: worker/src/index.ts
+Cost: zero. KV free tier is 100k reads/day, 1k writes/day.
+
+---
+
+## [x] Upgrade 3 — Apify Actor timeout increase  ← DONE 2026-03-19
+Change the Actor timeout from 300 seconds to 3600 seconds in the Apify UI.
+Apify console → Actor → Settings → Timeout → 3600
+Playwright-based tools (extract_reddit, extract_yc, extract_producthunt) need
+more than 5 minutes to launch Chromium and scrape. They will keep timing out
+until this is changed. This is a UI field change, not a code change.
+Cost: zero.
+
+---
+
+## [x] Upgrade 4 — D1 deduplication in the cron job  ← DONE (hash-based dedup already in runScheduledScrape)
+Before inserting a new scrape result, check if the same source_url was already
+stored in the last 24 hours. If yes, skip the insert.
+Prevents the scrape_results table from filling with duplicate data across
+consecutive cron runs, keeping the historical dataset clean for the intelligence
+layer (Layer 7 in the roadmap).
+Location: the cron job handler in the Worker code.
+Cost: zero.
+
+---
+
+## [x] Upgrade 5 — Structured JSON response form  ← DONE 2026-03-19
+Add the optional JSON form defined in FRESHCONTEXT_SPEC.md alongside the text
+envelope in every adapter response. The JSON form has: source_url, content_date,
+retrieved_at, freshness_confidence, adapter, freshness_score.
+When a request has Accept: application/json, serve the structured form.
+Both forms can be returned together — text for agents, JSON for programmatic use.
+Location: src/tools/freshnessStamp.ts (same file as Upgrade 1, do together)
+Cost: zero.
+
+---
+
+## [x] Upgrade 6 — GitHub Actions CI/CD automation  ← DONE 2026-03-19
+.github/workflows/publish.yml created. Triggers on every push to main.
+Runs npm ci → npm run build → npm publish using NPM_TOKEN secret.
+continue-on-error on publish so doc-only pushes don't fail the workflow.
+First run: green checkmark, 23 seconds.
+Manual PowerShell build/publish commands no longer needed.
+
+---
+
+## [x] Upgrade 7 — server.json version sync  ← DONE 2026-03-19
+server.json (MCP Registry listing) shows version 0.3.1 while package.json
+is at 0.3.10. Anyone discovering FreshContext via the MCP Registry sees an
+outdated version number. Fix by updating server.json manually now, then
+optionally add a workflow step that syncs the version automatically on each
+GitHub Actions run.
+Location: server.json — change "version" field to match package.json.
+Cost: zero.
+
+---
+
+## Priority order for remaining six upgrades
+
+Do Upgrade 3 first — it is one UI field change and immediately fixes the
+broken Apify Actor runs. Do Upgrades 1 and 5 together second since they
+both touch freshnessStamp.ts and completing them makes FreshContext fully
+spec-compliant. Do Upgrade 2 third — KV caching makes the Worker resilient
+against upstream API instability. Do Upgrade 4 fourth — D1 deduplication
+prepares the dataset for the future intelligence layer. Do Upgrade 7 last —
+a simple version number correction, low urgency but worth keeping clean.

package/ARCHITECTURE_UPGRADE_ROADMAP_V1.md

@@ -0,0 +1,174 @@
+# FreshContext — Architecture Upgrade Roadmap V1
+**Date:** 2026-03-19
+**Author:** Immanuel Gabriel (Prince Gabriel), Grootfontein, Namibia
+
+This document describes every free structural upgrade available to FreshContext,
+prioritised by impact, with implementation notes for each.
+
+---
+
+## Upgrade 1 — freshness_score numeric field (HIGHEST PRIORITY)
+
+**What it is:** The FreshContext Specification v1.0 defines an optional freshness_score
+field (0-100) calculated as: max(0, 100 - (days_since_retrieved * decay_rate)).
+Right now every response carries the text envelope and the confidence level (high/medium/low)
+but not the numeric score. This is the one remaining piece that makes FreshContext fully
+spec-compliant by your own standard.
+
+**Why it matters:** Once the score exists, agents can filter results programmatically —
+"only use results with freshness_score > 70" rather than parsing the string confidence
+level. This is the difference between a label and a query parameter. It also strengthens
+the acquisition narrative: the spec is complete, the reference implementation is complete,
+and the standard is fully self-consistent.
+
+**Domain-specific decay rates from the spec:**
+Financial data decays at 5.0 (half-life ~10 days). Job listings at 3.0 (~17 days).
+News and HN at 2.0 (~25 days). GitHub repos at 1.0 (~50 days). Academic papers at 0.3
+(~167 days). General web content defaults to 1.5.
+
+**Where to implement:** In src/tools/freshnessStamp.ts — the function that wraps every
+adapter result already has retrieved_at and content_date. Add a calculateFreshnessScore
+function that takes content_date, decay_rate (looked up by adapter name), and returns
+the numeric score. Add it to both the text envelope and the JSON form.
+
+**Cost:** Zero. Pure TypeScript logic, no new services.
+
+---
+
+## Upgrade 2 — Cloudflare KV response caching
+
+**What it is:** When the same query hits an adapter twice within a short window, the
+Worker currently makes two full upstream API calls. KV caching stores the first result
+with a TTL and serves subsequent identical requests from cache — meaning the upstream
+API (USASpending, GitHub, HN, etc.) only gets called once per cache window.
+
+**Why it matters:** This reduces the chance of hitting upstream rate limits, makes
+repeated queries near-instant for users, and reduces Worker CPU time. For adapters like
+extract_govcontracts that call a government API, caching also reduces the risk of
+temporary blocks from aggressive polling.
+
+**Implementation:** In the Worker code (worker/src/index.ts or equivalent), before calling
+the adapter, compute a cache key as sha256(tool + ":" + url). Call env.KV.get(cacheKey).
+If the result exists, return it immediately. If not, run the adapter, then call
+env.KV.put(cacheKey, result, { expirationTtl: ttl }) before returning. Use adapter-specific
+TTLs — 3600 seconds (1 hour) for HN and Reddit, 21600 (6 hours) for GitHub and YC,
+86400 (24 hours) for govcontracts and scholar.
+
+**Cost:** Zero. KV reads are free up to 100,000 per day, writes free up to 1,000 per day
+on Cloudflare's free tier. You are nowhere near those limits.
+
+---
+
+## Upgrade 3 — Apify Actor timeout increase
+
+**What it is:** The Apify Actor timeout is currently set to 300 seconds (5 minutes). Tools
+that use Playwright to launch a browser — extract_reddit, extract_yc, extract_producthunt —
+need more time than this to launch Chromium, navigate, wait for the page to render, and
+extract content. They will keep timing out until this setting is increased.
+
+**Where to change it:** Apify console → your Actor → Settings → Timeout. Change from
+300 to 3600 (1 hour). This is a UI change, not a code change.
+
+**Cost:** Zero. The timeout setting is just a number. You won't actually use anywhere
+near 3600 seconds — most tools complete in 10-30 seconds. The setting just prevents Apify
+from killing the process prematurely for the slower Playwright-based tools.
+
+---
+
+## Upgrade 4 — D1 deduplication in the cron job
+
+**What it is:** Every 6 hours the cron job runs all 18 watched queries and stores results
+in the scrape_results D1 table. Right now there is no deduplication — if the same article
+or repo appears in two consecutive cron runs, it gets stored twice. Over time this creates
+noise in the dataset and wastes storage.
+
+**Implementation:** Before inserting a new result, run a SELECT to check whether a row
+with the same source_url already exists within the last 24 hours. If it does, skip the
+insert. This is a single SQL WHERE clause addition to the existing insert logic.
+
+**Why it matters:** As you build the intelligence layer (Layer 7 in the roadmap), the
+quality of the historical signal depends on clean, deduplicated data. Starting deduplication
+now means the dataset is clean by the time you need it.
+
+**Cost:** Zero. D1 reads are free up to 25 million rows per day. A deduplication check
+adds one read per result per cron run — trivially within limits.
+
+---
+
+## Upgrade 5 — Structured JSON response form in every adapter
+
+**What it is:** The FreshContext Specification defines two valid response formats — the
+text envelope ([FRESHCONTEXT]...[/FRESHCONTEXT]) and an optional structured JSON form with
+a freshcontext object containing source_url, content_date, retrieved_at,
+freshness_confidence, adapter, and freshness_score fields. Right now only the text envelope
+is returned. Adding the JSON form makes FreshContext usable programmatically without
+parsing the text envelope.
+
+**Implementation:** In src/tools/freshnessStamp.ts, after assembling the text envelope,
+also return a structured object. When the Worker serves a response, detect whether the
+request has Accept: application/json and serve the structured form instead of the text
+form if so. Both formats can also be returned together — text for human/agent reading,
+JSON for programmatic use.
+
+**Cost:** Zero. This is a response format change, no new services.
+
+---
+
+## Upgrade 6 — GitHub Actions: version bump automation
+
+**What it is:** The current GitHub Actions workflow (publish.yml) runs npm publish on every
+push, but only succeeds if the version in package.json has changed. Right now you manually
+bump the version before pushing. A small addition to the workflow can automate this by
+running npm version patch automatically before the publish step — so every push to main
+creates a new patch version and publishes it without any manual intervention.
+
+**Tradeoff:** This means every push creates a new npm version, which may not always be
+desirable for documentation-only changes. A better approach is to only auto-bump when
+commits touch src/ or .actor/ — which can be detected in the workflow with a path filter.
+
+**Implementation:** Add a paths filter to the workflow trigger so it only runs the publish
+step when source files change. Then add an npm version patch --no-git-tag-version step
+before the publish step. Push the bumped package.json back to the repo using a
+git commit and git push within the workflow (requires GITHUB_TOKEN, which is automatically
+available in all Actions workflows at no cost).
+
+**Cost:** Zero.
+
+---
+
+## Upgrade 7 — server.json version sync check
+
+**What it is:** The server.json file (used by the MCP Registry listing) still shows version
+0.3.1 while package.json is at 0.3.10. This discrepancy means anyone who discovers
+FreshContext via the MCP Registry sees an outdated version number. It is a cosmetic issue
+but it affects credibility in a space where people are evaluating tools carefully.
+
+**Implementation:** Add a step to the GitHub Actions workflow that reads the version from
+package.json and uses sed or node -e to update the version field in server.json to match
+before committing. Alternatively, update server.json manually now and keep it in sync
+going forward.
+
+**Cost:** Zero.
+
+---
+
+## Priority Order for Implementation
+
+The order that maximises impact relative to effort is as follows. Implement the Apify
+timeout increase first because it is a one-field UI change that immediately fixes the
+broken Actor runs. Implement KV caching second because it makes the Worker more robust
+against upstream API instability and improves response times for repeat queries. Implement
+the freshness_score calculation third because it completes the spec and strengthens every
+conversation about acquisition or partnership. Implement D1 deduplication fourth because
+it improves data quality for the intelligence layer you will eventually build. Implement
+the structured JSON response form fifth as part of the same PR as freshness_score since
+they touch the same file. Implement the GitHub Actions version sync last as a quality-of-life
+automation.
+
+The total engineering cost of all six remaining upgrades is approximately 4-6 hours of
+focused work. All run entirely within free tiers.
+
+---
+
+*"The work isn't gone. It's just waiting to be continued."*
+*— Prince Gabriel, Grootfontein, Namibia*

package/HANDOFF.md

@@ -0,0 +1,184 @@
+# FreshContext — Handoff Document
+**Version:** 0.3.11
+**Date:** 2026-03-20
+**Author:** Immanuel Gabriel (Prince Gabriel), Grootfontein, Namibia
+**Contact:** gimmanuel73@gmail.com
+
+---
+
+## What You Are Receiving
+
+FreshContext is a web intelligence engine for AI agents. It wraps every piece of
+retrieved web data in a structured freshness envelope — exact retrieval timestamp,
+publication date estimate, freshness confidence (high/medium/low), and a 0-100
+numeric score with domain-specific decay rates.
+
+15 tools. No API keys required. Deployed globally on Cloudflare's edge. Listed on
+Anthropic's official MCP Registry, npm, and Apify Store.
+
+The two tools that exist nowhere else: extract_govcontracts (US federal contract
+intelligence via USASpending.gov) and extract_changelog (product release velocity
+from any GitHub repo, npm package, or website).
+
+---
+
+## Services and Infrastructure
+
+### 1. GitHub Repository
+URL: https://github.com/PrinceGabriel-lgtm/freshcontext-mcp
+Branch: main
+Transfer method: GitHub Settings > Transfer ownership
+What it contains: All source code, Dockerfile, specs, session saves, roadmap
+
+### 2. npm Package
+Package: freshcontext-mcp (v0.3.11)
+URL: https://www.npmjs.com/package/freshcontext-mcp
+Account: immanuel-gabriel on npmjs.com
+Transfer method: npm owner add new-username freshcontext-mcp
+Note: Published automatically via GitHub Actions on every push to main
+
+### 3. Cloudflare Account
+What lives here:
+  Worker:     freshcontext-mcp (the live MCP endpoint)
+  D1:         freshcontext-db (ID: d9898d65-f67e-4dcb-abdc-7f7b53f2d444)
+  KV:         RATE_LIMITER and CACHE (IDs in wrangler.jsonc)
+  Cron:       0 */6 * * * (every 6 hours, runs automatically)
+  Endpoint:   https://freshcontext-mcp.gimmanuel73.workers.dev/mcp
+
+Transfer method: Add new account as Super Administrator, remove original.
+D1 export: wrangler d1 export freshcontext-db --output=dump.sql
+
+### 4. Apify Actor
+Actor:        prince_gabriel/freshcontext-mcp
+URL:          https://apify.com/prince_gabriel/freshcontext-mcp
+Monetization: $50.00 per 1,000 results (Pay per event)
+Transfer method: Re-publish under new Apify account
+
+### 5. MCP Registry Listing
+Entry:  io.github.PrinceGabriel-lgtm/freshcontext
+Config: server.json in the GitHub repo
+Transfer method: Update server.json with new repo URL and re-submit
+
+### 6. GitHub Actions CI/CD
+File:   .github/workflows/publish.yml
+Action: On every push to main — npm ci > tsc > npm publish
+Secret: NPM_TOKEN (granular access token from npmjs.com, renew annually)
+
+---
+
+## Credentials Map (categories only — not values)
+
+| Credential | Where Used | Location |
+|---|---|---|
+| API_KEY | Worker auth header | Cloudflare env var |
+| ANTHROPIC_KEY | Synthesis/briefing endpoint | Cloudflare env var |
+| GITHUB_TOKEN | GitHub API rate limit bypass | Cloudflare env var |
+| NPM_TOKEN | GitHub Actions auto-publish | GitHub secret |
+
+---
+
+## Codebase Map
+
+src/server.ts          — MCP stdio server, all 15 tools registered here
+src/apify.ts           — Apify Actor entry point (read input, call adapter, exit)
+src/security.ts        — URL validation, SSRF prevention
+src/types.ts           — FreshContext, AdapterResult, ExtractOptions interfaces
+src/adapters/          — One file per data source (13 files)
+  changelog.ts         — UNIQUE: GitHub Releases API + npm + auto-discover
+  govcontracts.ts      — UNIQUE: USASpending.gov federal contract awards
+src/tools/
+  freshnessStamp.ts    — Score calculation, JSON form, text envelope
+worker/src/worker.ts   — Cloudflare Worker: 15 tools + KV cache + rate limit
+                         + D1 cron scraper + briefing formatter
+.actor/Dockerfile      — Apify build config (Node 20, runs dist/apify.js)
+.actor/actor.json      — Apify Actor metadata
+FRESHCONTEXT_SPEC.md   — The open standard (MIT license)
+ROADMAP.md             — 10-layer product vision
+server.json            — MCP Registry listing
+.github/workflows/
+  publish.yml          — GitHub Actions CI/CD
+
+---
+
+## The 15 Tools
+
+Standard (11):
+  extract_github, extract_hackernews, extract_scholar, extract_arxiv,
+  extract_reddit, extract_yc, extract_producthunt, search_repos,
+  package_trends, extract_finance, search_jobs
+
+Composite landscapes (3):
+  extract_landscape      — YC + GitHub + HN + Reddit + Product Hunt + npm
+  extract_gov_landscape  — govcontracts + HN + GitHub + changelog
+  extract_finance_landscape — finance + HN + Reddit + GitHub + changelog
+
+Unique — not in any other MCP server (2):
+  extract_changelog    — release history from any repo, package, or website
+  extract_govcontracts — US federal contract awards from USASpending.gov
+
+---
+
+## D1 Database Schema
+
+watched_queries  — 18 active monitored topics
+  id, adapter, query, label, filters, enabled, last_run_at
+
+scrape_results   — raw results, deduplicated by content hash
+  id, watched_query_id, adapter, query, raw_content, result_hash, is_new, scraped_at
+
+briefings        — formatted intelligence reports per cron run
+  id, user_id, summary, new_results_count, adapters_run, created_at
+
+user_profiles    — personalization data for briefing synthesis
+  id, name, skills, certifications, targets, location, context
+
+---
+
+## The FreshContext Specification
+
+FRESHCONTEXT_SPEC.md is the open standard, MIT license, authored March 2026.
+Any implementation returning the [FRESHCONTEXT]...[/FRESHCONTEXT] envelope
+or the structured JSON form with freshcontext.retrieved_at and
+freshcontext.freshness_confidence is FreshContext-compatible.
+
+The spec is the durable asset. The code is the reference implementation.
+
+---
+
+## What Keeps Running Without You
+
+The Cloudflare cron fires every 6 hours automatically. Every run scrapes all 18
+watched queries, deduplicates by content hash, stores new signals in D1, and
+generates a briefing. The dataset accumulates indefinitely. No action required.
+
+---
+
+## Pending Items at Time of Handoff
+
+Apify Playwright — extract_reddit, extract_yc, extract_producthunt may error on
+Apify. Fix: add RUN npx playwright install chromium --with-deps to .actor/Dockerfile.
+
+Synthesis endpoint — /briefing/now is paused. Needs ANTHROPIC_KEY in Worker env
+and credits loaded at console.anthropic.com. Infrastructure is fully built.
+
+Agnost AI analytics — free MCP analytics offered by Apify. Sign up at app.agnost.ai,
+add one line to server.ts. Gives tool call tracking and usage dashboards.
+
+Apify GitHub Actions trigger — npm publish is automated but Apify rebuild is
+manual. Add a workflow step calling Apify's API to auto-rebuild on push.
+
+---
+
+## Outreach Status at Time of Handoff
+
+19 confirmed delivered emails across US, Singapore, China, and Europe.
+Cloudflare for Startups — awaiting reply (10 business day window).
+GovTech Singapore — auto-acknowledged, reply expected by March 25, 2026.
+All others — first contact sent, no replies yet.
+
+HN post live: https://news.ycombinator.com/user?id=Prince-Gabriel
+
+---
+
+*"The work isn't gone. It's just waiting to be continued."*
+*— Prince Gabriel, Grootfontein, Namibia*