footnote-mcp 1.0.0__tar.gz

footnote_mcp-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 KazKozDev
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

footnote_mcp-1.0.0/PKG-INFO

@@ -0,0 +1,286 @@
+Metadata-Version: 2.4
+Name: footnote-mcp
+Version: 1.0.0
+Summary: MCP server for source-grounded web research: search, structured-data extraction, evidence verification, and browser automation.
+License-Expression: MIT
+Keywords: mcp,model-context-protocol,web-research,scraping,fact-checking,browser-automation
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: mcp>=1.0.0
+Requires-Dist: ollama>=0.4
+Requires-Dist: playwright>=1.49
+Requires-Dist: curl_cffi>=0.5.10
+Requires-Dist: beautifulsoup4>=4.12
+Requires-Dist: trafilatura>=1.12
+Requires-Dist: rank-bm25>=0.2
+Requires-Dist: openpyxl>=3.1
+Requires-Dist: pypdf>=4.0
+Requires-Dist: xlrd>=2.0
+Requires-Dist: pdfplumber>=0.11
+Requires-Dist: pytesseract>=0.3
+Requires-Dist: holidays>=0.60
+Dynamic: license-file
+
+<p align="center">
+  <img src=".github/banner.png" alt="footnote-mcp" width="350">
+</p>
+
+An MCP server for source-grounded web research. It searches the web, fetches and
+extracts pages, pulls structured data out of tables/files/APIs, and — the part that
+sets it apart — **verifies that a claim is actually supported by its source** instead of
+trusting a snippet. 42 tools over stdio MCP, driven by any MCP client (Claude Desktop,
+Cursor) or by the companion [Scholiast](https://github.com/KazKozDev/scholiast) research agent.
+
+The design priority is *trustworthiness over convenience*: search snippets are treated as
+discovery only, every fetched page is cached with provenance, and claims are checked
+against the source text before they count. It also degrades gracefully — with no API keys
+and no config it still works (scraped search + an automatic headless-browser fallback +
+an offline verification heuristic); keys and env vars only make it better.
+
+## Quick start
+
+From source (Python ≥ 3.10):
+
+```bash
+python3 -m venv .venv && source .venv/bin/activate
+pip install -e .                        # installs the `footnote-mcp` console script + deps
+python -m playwright install chromium   # the headless browser used by the fetch fallback
+footnote-mcp                            # start the server (speaks MCP over stdio)
+```
+
+`footnote-mcp` now waits for an MCP client on stdio. Point a client at it by dropping this
+into its MCP settings (Claude Desktop: `claude_desktop_config.json`; Cursor: `~/.cursor/mcp.json`):
+
+```json
+{
+  "mcpServers": {
+    "footnote": { "command": "footnote-mcp" }
+  }
+}
+```
+
+No API keys are required to start — search falls back to scraping Bing + DuckDuckGo. Add
+keys later under `"env"` (see [Search backends](#search-backends)). Pass `--headed` to watch
+the browser tier work.
+
+To run without installing, straight from the source tree:
+
+```bash
+PYTHONPATH=src python -m footnote_mcp
+```
+
+## Verifying claims — the differentiator
+
+The reason to use this over a plain search tool is `evidence_entailment` and friends:
+they tell a claim a source *supports* from one it does not. `benchmarks/run_benchmark.py`
+measures that on a labeled set of claim/source pairs (and demos `corroborate_claim` and
+`locate_claim_span`):
+
+```bash
+python benchmarks/run_benchmark.py                    # offline heuristic (deterministic)
+python benchmarks/run_benchmark.py --backend ollama   # LLM judge (needs ollama)
+```
+
+Offline-heuristic result on the labeled set ([benchmarks/REPORT.md](benchmarks/REPORT.md)):
+
+| Set | n | Accuracy | Unsupported-claim catch rate | Precision on "supported" |
+|-----|---|----------|------------------------------|--------------------------|
+| Data domain (numeric + factual) | 15 | 100% | 100% | 100% |
+| Overall (incl. semantic) | 18 | 83% | 78% | 80% |
+
+On its design domain — numeric and factual data claims — the offline heuristic never
+blesses an unsupported claim and never misses one. Its blind spot is purely-semantic
+negation/paraphrase; for those, `evidence_entailment` with `backend="ollama"` (a local LLM
+judge) closes the gap. Run the `--backend ollama` line above to score that path on your own
+machine.
+
+## Tool surface (42 tools)
+
+<details>
+<summary><b>Discovery and reading</b> (9 tools)</summary>
+
+| Tool | Description |
+|------|-------------|
+| `web_search` | Keyed provider (Tavily/Brave/Google) when available, else scraped Bing + DuckDuckGo. Snippets are discovery only. |
+| `web_search_recent` | Search restricted to a recency window (day/week/month/year). |
+| `web_deep_search` | Search, fetch, extract, rerank, and return source context. |
+| `web_read` | Fetch one URL, extract text, classify source quality, persist cache metadata. |
+| `scholarly_search` | Search arXiv (papers) or Wikipedia (encyclopedic) corpora. |
+| `web_archive_fetch` | Find the closest Wayback Machine snapshot for a dead/changed URL. |
+| `web_fetch_authenticated` | Fetch a page that needs cookies or custom headers. |
+| `web_crawl` | Breadth-first crawl from a start URL, on-host by default (≤ 50 pages). |
+| `generate_search_queries` | Generate operator queries (`site:`, `filetype:csv`, API/data-table variants). |
+
+</details>
+
+<details>
+<summary><b>Structured data</b> (9 tools)</summary>
+
+| Tool | Description |
+|------|-------------|
+| `web_extract_tables` | Parse HTML tables into `columns`/`rows` with source-URL provenance. |
+| `web_detect_downloads` | Detect linked CSV/TSV/XLS/XLSX/PDF/JSON/XML files. |
+| `web_parse_file` | Download and parse CSV/TSV/XLS/XLSX/PDF/JSON. |
+| `web_fetch_json` | Fetch direct API/JSON endpoints into parsed JSON. |
+| `check_date_completeness` | Validate required date coverage (day/week/month). |
+| `resolve_units` | Detect currencies, currency pairs, measurement units. |
+| `validate_unit_rows` | Reject rows with incompatible units or currency pairs. |
+| `reconcile_time_series` | Align series on a key, compute deltas, flag missing keys/outliers. |
+| `export_dataset` | Write consolidated rows to a `csv`/`xlsx`/`json` file. |
+
+</details>
+
+<details>
+<summary><b>Source quality and verification</b> (8 tools)</summary>
+
+| Tool | Description |
+|------|-------------|
+| `classify_source` | Classify official / aggregator / blog / forum / interactive / blocked / error. |
+| `evidence_entailment` | Strict claim-vs-source checker: `heuristic`, `auto`, `ollama`, optional `local_nli`. |
+| `corroborate_claim` | Triangulate a claim across excerpts (corroborated / conflicting / single_source / …). |
+| `locate_claim_span` | Locate supporting sentence(s) with char offsets and a containment score. |
+| `source_cache_get` / `source_cache_put` | Inspect and write persistent source-cache entries. |
+| `build_research_debug_report` | Compact report of queries, URLs, source quality, verification gaps. |
+| `startup_health_check` | Check parser, OCR, browser, and cache dependencies. |
+
+</details>
+
+<details>
+<summary><b>Controlled extraction recipes</b> (6 tools)</summary>
+
+When generic parsers fail, synthesize a sandboxed parser:
+
+| Tool | Description |
+|------|-------------|
+| `tool_spec_propose` | Propose a task-specific extraction recipe spec. |
+| `tool_code_generate` | Generate a starter `extract(source_text, input_payload)` recipe. |
+| `tool_code_validate` | Validate recipe code against a static safety allowlist. |
+| `tool_code_run_sandboxed` | Run validated code in a limited subprocess (JSON output only). |
+| `tool_promote` | Save a validated recipe as reusable memory (no server edit). |
+| `recipe_registry` | Manage promoted recipes: `list` / `get` / `run` / `delete`. |
+
+</details>
+
+<details>
+<summary><b>Browser fallback</b> (10 tools)</summary>
+
+A controlled Chromium session for JS-heavy or interactive pages:
+
+| Tool | Description |
+|------|-------------|
+| `web_navigate` · `web_snapshot` · `web_click` · `web_type` · `web_extract` · `web_scroll` | Drive a page via stable element refs. |
+| `browser_set_date_range` · `browser_extract_tables` · `browser_extract_tables_for_date_range` | Set a date range, submit, extract visible tables. |
+| `web_screenshot` | Save a PNG and optionally OCR text locked inside the image. |
+
+</details>
+
+## Search backends
+
+`web_search` (and everything built on it) routes through a provider layer. With an API key
+it uses that provider; otherwise it scrapes Bing + DuckDuckGo. Results are normalized to one
+shape regardless of backend.
+
+| Provider | Env vars | Notes |
+|----------|----------|-------|
+| Tavily | `TAVILY_API_KEY` | LLM-oriented search API. |
+| Brave | `BRAVE_API_KEY` | Independent web index. |
+| Google | `GOOGLE_API_KEY` + `GOOGLE_CSE_ID` | Programmable Search (Custom Search JSON API). |
+| Bing + DuckDuckGo | none | Default fallback; scraped, no key. |
+
+`auto` (default) tries each keyed provider in order Tavily → Brave → Google, then scrapes.
+Force one with the `provider` argument (`tavily`/`brave`/`google`/`scrape`).
+
+**Semantic reranking.** Pass `semantic: true` to `web_search` to reorder by meaning rather
+than keyword overlap: it over-fetches, embeds query and results with a local ollama model,
+and sorts by cosine similarity (each result gains `semantic_score`). Best-effort — if ollama
+is unavailable the original order is returned. Model: `FOOTNOTE_EMBED_MODEL` (default `bge-m3`).
+
+## Fetching & anti-bot ladder
+
+`web_read` fetches through an escalation ladder ([scraper.py](src/footnote_mcp/scraper.py)):
+the cheapest method runs first and escalates only when a result looks blocked or empty. A
+block/quality detector decides when to escalate; a per-domain rate limiter, circuit breaker,
+and negative cache keep it polite. The tier used and the full attempt trace come back in
+`fetch_tier` / `scrape_tiers`.
+
+| Tier | Method | Enabled by |
+|------|--------|-----------|
+| 1 | HTTP (curl_cffi TLS impersonation) | always |
+| 2 | HTTP through a rotating proxy | `FOOTNOTE_PROXIES` set |
+| 3 | Headless Chromium (runs JavaScript) | `FOOTNOTE_BROWSER_FALLBACK=1` (default on) |
+| 4 | Chromium through a proxy | proxies + browser |
+| 5 | Hosted scrape API (Firecrawl / ScrapingBee) | `FOOTNOTE_SCRAPE_API` set |
+
+With nothing configured it is the plain HTTP path plus an automatic browser fallback for
+JavaScript-rendered pages.
+
+| Env var | Default | Purpose |
+|---------|---------|---------|
+| `FOOTNOTE_BROWSER_FALLBACK` | `1` | Escalate blocked/JS pages to headless Chromium. |
+| `FOOTNOTE_PROXIES` | _(none)_ | Comma-separated proxy URLs; sticky per domain with health tracking. |
+| `FOOTNOTE_SCRAPE_API` | _(none)_ | `firecrawl` or `scrapingbee` (needs the matching API key). |
+| `FOOTNOTE_DOMAIN_RPS` / `_BURST` | `3` / `5` | Per-domain rate limit (token bucket). |
+| `FOOTNOTE_BREAKER_THRESHOLD` / `_COOLDOWN` | `5` / `120` | Per-domain circuit breaker. |
+| `FOOTNOTE_NEGCACHE_TTL` | `300` | Seconds to remember a blocked URL. |
+| `FOOTNOTE_THIN_CONTENT_CHARS` | `200` | Below this extracted length, a script-heavy page counts as a JS shell. |
+
+## Runtime data
+
+```text
+~/.footnote-mcp/source_cache/        # persistent page cache (with provenance)
+~/.footnote-mcp/research_memory.json # persistent research memory
+```
+
+Override the cache location with `FOOTNOTE_SOURCE_CACHE=/path/to/cache footnote-mcp`.
+
+`check_date_completeness` supports the calendars `calendar`, `business_day`, `crypto_24_7`,
+`forex_weekday`, `us_business_day`, and `ru_business_day` (pass explicit `holidays` for
+source-specific ones; the `us_`/`ru_` variants use the optional `holidays` package).
+
+## Other install paths
+
+**Docker** bundles Chromium and tesseract — nothing else to install:
+
+```bash
+docker build -t footnote-mcp .
+docker run -i --rm footnote-mcp        # the client launches this; see MCP config below
+```
+
+```json
+{ "mcpServers": { "footnote": { "command": "docker", "args": ["run", "-i", "--rm", "footnote-mcp"] } } }
+```
+
+**pipx / uvx** (isolated install of the entry point):
+
+```bash
+pipx install /path/to/footnote-mcp          # or: pipx install git+<repo-url>
+uvx --from /path/to/footnote-mcp footnote-mcp   # ad-hoc, no install
+```
+
+**OCR.** PDF/image OCR uses `pytesseract` + the system `tesseract` binary (`brew install
+tesseract` on macOS). **Local NLI backend** for `evidence_entailment` `backend="local_nli"`:
+`pip install -r requirements-nli.txt` (model via `FOOTNOTE_NLI_MODEL`). Either way,
+`startup_health_check` reports what is actually available. Runtime dependency ranges
+are declared in `pyproject.toml` and mirrored in `requirements.txt`.
+
+## Tests
+
+```bash
+pip install -r requirements-dev.txt
+python -m pytest -q          # offline unit + smoke tests; no network or keys needed
+```
+
+`tests/test_mcp_smoke.py` launches the server over real MCP stdio and exercises the tools
+end to end against a local HTTP fixture; the rest are offline unit tests of the parsers,
+fetch ladder, search providers, and dispatch. The live search test is opt-in:
+
+```bash
+RUN_LIVE_WEB_TESTS=1 python -m pytest -m live
+```
+
+CI runs the same suite (`.github/workflows/tests.yml`).
+
+## License
+
+MIT — see [LICENSE](LICENSE).

footnote_mcp-1.0.0/README.md

@@ -0,0 +1,262 @@
+<p align="center">
+  <img src=".github/banner.png" alt="footnote-mcp" width="350">
+</p>
+
+An MCP server for source-grounded web research. It searches the web, fetches and
+extracts pages, pulls structured data out of tables/files/APIs, and — the part that
+sets it apart — **verifies that a claim is actually supported by its source** instead of
+trusting a snippet. 42 tools over stdio MCP, driven by any MCP client (Claude Desktop,
+Cursor) or by the companion [Scholiast](https://github.com/KazKozDev/scholiast) research agent.
+
+The design priority is *trustworthiness over convenience*: search snippets are treated as
+discovery only, every fetched page is cached with provenance, and claims are checked
+against the source text before they count. It also degrades gracefully — with no API keys
+and no config it still works (scraped search + an automatic headless-browser fallback +
+an offline verification heuristic); keys and env vars only make it better.
+
+## Quick start
+
+From source (Python ≥ 3.10):
+
+```bash
+python3 -m venv .venv && source .venv/bin/activate
+pip install -e .                        # installs the `footnote-mcp` console script + deps
+python -m playwright install chromium   # the headless browser used by the fetch fallback
+footnote-mcp                            # start the server (speaks MCP over stdio)
+```
+
+`footnote-mcp` now waits for an MCP client on stdio. Point a client at it by dropping this
+into its MCP settings (Claude Desktop: `claude_desktop_config.json`; Cursor: `~/.cursor/mcp.json`):
+
+```json
+{
+  "mcpServers": {
+    "footnote": { "command": "footnote-mcp" }
+  }
+}
+```
+
+No API keys are required to start — search falls back to scraping Bing + DuckDuckGo. Add
+keys later under `"env"` (see [Search backends](#search-backends)). Pass `--headed` to watch
+the browser tier work.
+
+To run without installing, straight from the source tree:
+
+```bash
+PYTHONPATH=src python -m footnote_mcp
+```
+
+## Verifying claims — the differentiator
+
+The reason to use this over a plain search tool is `evidence_entailment` and friends:
+they tell a claim a source *supports* from one it does not. `benchmarks/run_benchmark.py`
+measures that on a labeled set of claim/source pairs (and demos `corroborate_claim` and
+`locate_claim_span`):
+
+```bash
+python benchmarks/run_benchmark.py                    # offline heuristic (deterministic)
+python benchmarks/run_benchmark.py --backend ollama   # LLM judge (needs ollama)
+```
+
+Offline-heuristic result on the labeled set ([benchmarks/REPORT.md](benchmarks/REPORT.md)):
+
+| Set | n | Accuracy | Unsupported-claim catch rate | Precision on "supported" |
+|-----|---|----------|------------------------------|--------------------------|
+| Data domain (numeric + factual) | 15 | 100% | 100% | 100% |
+| Overall (incl. semantic) | 18 | 83% | 78% | 80% |
+
+On its design domain — numeric and factual data claims — the offline heuristic never
+blesses an unsupported claim and never misses one. Its blind spot is purely-semantic
+negation/paraphrase; for those, `evidence_entailment` with `backend="ollama"` (a local LLM
+judge) closes the gap. Run the `--backend ollama` line above to score that path on your own
+machine.
+
+## Tool surface (42 tools)
+
+<details>
+<summary><b>Discovery and reading</b> (9 tools)</summary>
+
+| Tool | Description |
+|------|-------------|
+| `web_search` | Keyed provider (Tavily/Brave/Google) when available, else scraped Bing + DuckDuckGo. Snippets are discovery only. |
+| `web_search_recent` | Search restricted to a recency window (day/week/month/year). |
+| `web_deep_search` | Search, fetch, extract, rerank, and return source context. |
+| `web_read` | Fetch one URL, extract text, classify source quality, persist cache metadata. |
+| `scholarly_search` | Search arXiv (papers) or Wikipedia (encyclopedic) corpora. |
+| `web_archive_fetch` | Find the closest Wayback Machine snapshot for a dead/changed URL. |
+| `web_fetch_authenticated` | Fetch a page that needs cookies or custom headers. |
+| `web_crawl` | Breadth-first crawl from a start URL, on-host by default (≤ 50 pages). |
+| `generate_search_queries` | Generate operator queries (`site:`, `filetype:csv`, API/data-table variants). |
+
+</details>
+
+<details>
+<summary><b>Structured data</b> (9 tools)</summary>
+
+| Tool | Description |
+|------|-------------|
+| `web_extract_tables` | Parse HTML tables into `columns`/`rows` with source-URL provenance. |
+| `web_detect_downloads` | Detect linked CSV/TSV/XLS/XLSX/PDF/JSON/XML files. |
+| `web_parse_file` | Download and parse CSV/TSV/XLS/XLSX/PDF/JSON. |
+| `web_fetch_json` | Fetch direct API/JSON endpoints into parsed JSON. |
+| `check_date_completeness` | Validate required date coverage (day/week/month). |
+| `resolve_units` | Detect currencies, currency pairs, measurement units. |
+| `validate_unit_rows` | Reject rows with incompatible units or currency pairs. |
+| `reconcile_time_series` | Align series on a key, compute deltas, flag missing keys/outliers. |
+| `export_dataset` | Write consolidated rows to a `csv`/`xlsx`/`json` file. |
+
+</details>
+
+<details>
+<summary><b>Source quality and verification</b> (8 tools)</summary>
+
+| Tool | Description |
+|------|-------------|
+| `classify_source` | Classify official / aggregator / blog / forum / interactive / blocked / error. |
+| `evidence_entailment` | Strict claim-vs-source checker: `heuristic`, `auto`, `ollama`, optional `local_nli`. |
+| `corroborate_claim` | Triangulate a claim across excerpts (corroborated / conflicting / single_source / …). |
+| `locate_claim_span` | Locate supporting sentence(s) with char offsets and a containment score. |
+| `source_cache_get` / `source_cache_put` | Inspect and write persistent source-cache entries. |
+| `build_research_debug_report` | Compact report of queries, URLs, source quality, verification gaps. |
+| `startup_health_check` | Check parser, OCR, browser, and cache dependencies. |
+
+</details>
+
+<details>
+<summary><b>Controlled extraction recipes</b> (6 tools)</summary>
+
+When generic parsers fail, synthesize a sandboxed parser:
+
+| Tool | Description |
+|------|-------------|
+| `tool_spec_propose` | Propose a task-specific extraction recipe spec. |
+| `tool_code_generate` | Generate a starter `extract(source_text, input_payload)` recipe. |
+| `tool_code_validate` | Validate recipe code against a static safety allowlist. |
+| `tool_code_run_sandboxed` | Run validated code in a limited subprocess (JSON output only). |
+| `tool_promote` | Save a validated recipe as reusable memory (no server edit). |
+| `recipe_registry` | Manage promoted recipes: `list` / `get` / `run` / `delete`. |
+
+</details>
+
+<details>
+<summary><b>Browser fallback</b> (10 tools)</summary>
+
+A controlled Chromium session for JS-heavy or interactive pages:
+
+| Tool | Description |
+|------|-------------|
+| `web_navigate` · `web_snapshot` · `web_click` · `web_type` · `web_extract` · `web_scroll` | Drive a page via stable element refs. |
+| `browser_set_date_range` · `browser_extract_tables` · `browser_extract_tables_for_date_range` | Set a date range, submit, extract visible tables. |
+| `web_screenshot` | Save a PNG and optionally OCR text locked inside the image. |
+
+</details>
+
+## Search backends
+
+`web_search` (and everything built on it) routes through a provider layer. With an API key
+it uses that provider; otherwise it scrapes Bing + DuckDuckGo. Results are normalized to one
+shape regardless of backend.
+
+| Provider | Env vars | Notes |
+|----------|----------|-------|
+| Tavily | `TAVILY_API_KEY` | LLM-oriented search API. |
+| Brave | `BRAVE_API_KEY` | Independent web index. |
+| Google | `GOOGLE_API_KEY` + `GOOGLE_CSE_ID` | Programmable Search (Custom Search JSON API). |
+| Bing + DuckDuckGo | none | Default fallback; scraped, no key. |
+
+`auto` (default) tries each keyed provider in order Tavily → Brave → Google, then scrapes.
+Force one with the `provider` argument (`tavily`/`brave`/`google`/`scrape`).
+
+**Semantic reranking.** Pass `semantic: true` to `web_search` to reorder by meaning rather
+than keyword overlap: it over-fetches, embeds query and results with a local ollama model,
+and sorts by cosine similarity (each result gains `semantic_score`). Best-effort — if ollama
+is unavailable the original order is returned. Model: `FOOTNOTE_EMBED_MODEL` (default `bge-m3`).
+
+## Fetching & anti-bot ladder
+
+`web_read` fetches through an escalation ladder ([scraper.py](src/footnote_mcp/scraper.py)):
+the cheapest method runs first and escalates only when a result looks blocked or empty. A
+block/quality detector decides when to escalate; a per-domain rate limiter, circuit breaker,
+and negative cache keep it polite. The tier used and the full attempt trace come back in
+`fetch_tier` / `scrape_tiers`.
+
+| Tier | Method | Enabled by |
+|------|--------|-----------|
+| 1 | HTTP (curl_cffi TLS impersonation) | always |
+| 2 | HTTP through a rotating proxy | `FOOTNOTE_PROXIES` set |
+| 3 | Headless Chromium (runs JavaScript) | `FOOTNOTE_BROWSER_FALLBACK=1` (default on) |
+| 4 | Chromium through a proxy | proxies + browser |
+| 5 | Hosted scrape API (Firecrawl / ScrapingBee) | `FOOTNOTE_SCRAPE_API` set |
+
+With nothing configured it is the plain HTTP path plus an automatic browser fallback for
+JavaScript-rendered pages.
+
+| Env var | Default | Purpose |
+|---------|---------|---------|
+| `FOOTNOTE_BROWSER_FALLBACK` | `1` | Escalate blocked/JS pages to headless Chromium. |
+| `FOOTNOTE_PROXIES` | _(none)_ | Comma-separated proxy URLs; sticky per domain with health tracking. |
+| `FOOTNOTE_SCRAPE_API` | _(none)_ | `firecrawl` or `scrapingbee` (needs the matching API key). |
+| `FOOTNOTE_DOMAIN_RPS` / `_BURST` | `3` / `5` | Per-domain rate limit (token bucket). |
+| `FOOTNOTE_BREAKER_THRESHOLD` / `_COOLDOWN` | `5` / `120` | Per-domain circuit breaker. |
+| `FOOTNOTE_NEGCACHE_TTL` | `300` | Seconds to remember a blocked URL. |
+| `FOOTNOTE_THIN_CONTENT_CHARS` | `200` | Below this extracted length, a script-heavy page counts as a JS shell. |
+
+## Runtime data
+
+```text
+~/.footnote-mcp/source_cache/        # persistent page cache (with provenance)
+~/.footnote-mcp/research_memory.json # persistent research memory
+```
+
+Override the cache location with `FOOTNOTE_SOURCE_CACHE=/path/to/cache footnote-mcp`.
+
+`check_date_completeness` supports the calendars `calendar`, `business_day`, `crypto_24_7`,
+`forex_weekday`, `us_business_day`, and `ru_business_day` (pass explicit `holidays` for
+source-specific ones; the `us_`/`ru_` variants use the optional `holidays` package).
+
+## Other install paths
+
+**Docker** bundles Chromium and tesseract — nothing else to install:
+
+```bash
+docker build -t footnote-mcp .
+docker run -i --rm footnote-mcp        # the client launches this; see MCP config below
+```
+
+```json
+{ "mcpServers": { "footnote": { "command": "docker", "args": ["run", "-i", "--rm", "footnote-mcp"] } } }
+```
+
+**pipx / uvx** (isolated install of the entry point):
+
+```bash
+pipx install /path/to/footnote-mcp          # or: pipx install git+<repo-url>
+uvx --from /path/to/footnote-mcp footnote-mcp   # ad-hoc, no install
+```
+
+**OCR.** PDF/image OCR uses `pytesseract` + the system `tesseract` binary (`brew install
+tesseract` on macOS). **Local NLI backend** for `evidence_entailment` `backend="local_nli"`:
+`pip install -r requirements-nli.txt` (model via `FOOTNOTE_NLI_MODEL`). Either way,
+`startup_health_check` reports what is actually available. Runtime dependency ranges
+are declared in `pyproject.toml` and mirrored in `requirements.txt`.
+
+## Tests
+
+```bash
+pip install -r requirements-dev.txt
+python -m pytest -q          # offline unit + smoke tests; no network or keys needed
+```
+
+`tests/test_mcp_smoke.py` launches the server over real MCP stdio and exercises the tools
+end to end against a local HTTP fixture; the rest are offline unit tests of the parsers,
+fetch ladder, search providers, and dispatch. The live search test is opt-in:
+
+```bash
+RUN_LIVE_WEB_TESTS=1 python -m pytest -m live
+```
+
+CI runs the same suite (`.github/workflows/tests.yml`).
+
+## License
+
+MIT — see [LICENSE](LICENSE).

footnote_mcp-1.0.0/pyproject.toml

@@ -0,0 +1,49 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "footnote-mcp"
+version = "1.0.0"
+description = "MCP server for source-grounded web research: search, structured-data extraction, evidence verification, and browser automation."
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+keywords = ["mcp", "model-context-protocol", "web-research", "scraping", "fact-checking", "browser-automation"]
+
+dependencies = [
+    "mcp>=1.0.0",
+    "ollama>=0.4",
+    "playwright>=1.49",
+    "curl_cffi>=0.5.10",
+    "beautifulsoup4>=4.12",
+    "trafilatura>=1.12",
+    "rank-bm25>=0.2",
+    "openpyxl>=3.1",
+    "pypdf>=4.0",
+    "xlrd>=2.0",
+    "pdfplumber>=0.11",
+    "pytesseract>=0.3",
+    "holidays>=0.60",
+]
+
+[project.scripts]
+footnote-mcp = "footnote_mcp:cli"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src"]
+asyncio_default_fixture_loop_scope = "function"
+markers = [
+    "live: requires external network/search engines",
+]
+filterwarnings = [
+    "ignore:builtin type SwigPyPacked has no __module__ attribute:DeprecationWarning",
+    "ignore:builtin type SwigPyObject has no __module__ attribute:DeprecationWarning",
+    "ignore:builtin type swigvarlink has no __module__ attribute:DeprecationWarning",
+    "ignore:`torch\\.jit\\.interface` is deprecated.*:DeprecationWarning",
+]

footnote_mcp-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

footnote_mcp-1.0.0/src/footnote_mcp/__init__.py

@@ -0,0 +1,13 @@
+"""footnote-mcp: an MCP server for source-grounded web research.
+
+The runtime lives as submodules of this package (``server``, ``core``,
+``search``, ...). ``cli`` is the console-script entry point; the import of the
+heavy ``server`` module is deferred into the call so ``import footnote_mcp`` stays
+cheap.
+"""
+
+
+def cli():
+    from .server import cli as _cli
+
+    _cli()

footnote_mcp-1.0.0/src/footnote_mcp/__main__.py

@@ -0,0 +1,6 @@
+"""Allow ``python -m footnote_mcp`` to launch the server."""
+
+from . import cli
+
+if __name__ == "__main__":
+    cli()