deepresearch-flow 0.7.0__tar.gz → 0.7.2__tar.gz

{deepresearch_flow-0.7.0/python/deepresearch_flow.egg-info → deepresearch_flow-0.7.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: deepresearch-flow
-Version: 0.7.0
+Version: 0.7.2
 Summary: Workflow tools for paper extraction, review, and research automation.
 Author-email: DengQi <dengqi935@gmail.com>
 License: MIT License
@@ -520,7 +520,14 @@ server {
   }
 
   location /api/ {
-    proxy_pass http://127.0.0.1:8001/;
+    proxy_pass http://127.0.0.1:8001;
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+  }
+
+  location ^~ /mcp {
+    proxy_pass http://127.0.0.1:8001;
     proxy_set_header Host $host;
     proxy_set_header X-Real-IP $remote_addr;
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
@@ -555,6 +562,147 @@ uv run deepresearch-flow paper db api serve \
   --host 0.0.0.0 --port 8001
 ```
 
+### 3.1) MCP (FastMCP Streamable HTTP)
+
+This project exposes an MCP server mounted at `/mcp` on the snapshot API:
+
+- Endpoint: `http://<host>:8001/mcp` (same host/port as `paper db api serve`)
+- Transport: Streamable HTTP via `POST` only (no SSE; `GET` returns 405)
+- Protocol header: optional `mcp-protocol-version` (`2025-03-26` or `2025-06-18`)
+- Static reads: summary/source/translation are served as **text content** by reading snapshot static assets (local-first via `PAPER_DB_STATIC_EXPORT_DIR`, HTTP fallback via `PAPER_DB_STATIC_BASE` / `PAPER_DB_STATIC_BASE_URL`)
+
+Optional (avoid HTTP fetch by reading exported assets directly on the API host):
+
+```bash
+export PAPER_DB_STATIC_EXPORT_DIR=/data/paper-static
+```
+
+#### MCP Tools (API functions)
+
+<details>
+<summary><strong>search_papers(query, limit=10)</strong> — full-text search (relevance-ranked)</summary>
+
+- Args:
+  - `query` (str): keywords / topic query
+  - `limit` (int): number of results (clamped to API max page size)
+- Returns: list of `{ paper_id, title, year, venue, snippet_markdown }`
+
+</details>
+
+<details>
+<summary><strong>search_papers_by_keyword(keyword, limit=10)</strong> — facet keyword search</summary>
+
+- Args:
+  - `keyword` (str): keyword substring
+  - `limit` (int): number of results (clamped)
+- Returns: list of `{ paper_id, title, year, venue, snippet_markdown }`
+
+</details>
+
+<details>
+<summary><strong>get_paper_metadata(paper_id)</strong> — metadata + available summary templates</summary>
+
+- Args:
+  - `paper_id` (str)
+- Returns: dict with:
+  - `paper_id`, `title`, `year`, `venue`
+  - `doi`, `arxiv_id`, `openreview_id`, `paper_pw_url`
+  - `preferred_summary_template`, `available_summary_templates`
+
+</details>
+
+<details>
+<summary><strong>get_paper_summary(paper_id, template=None, max_chars=None)</strong> — summary JSON as raw text</summary>
+
+- Notes:
+  - Uses `preferred_summary_template` if `template` is omitted
+  - Returns the **full JSON content** (not a URL)
+- Args:
+  - `paper_id` (str)
+  - `template` (str | null)
+  - `max_chars` (int | null): truncation limit
+- Returns: JSON string (may include a `[truncated: ...]` marker)
+
+</details>
+
+<details>
+<summary><strong>get_paper_source(paper_id, max_chars=None)</strong> — source markdown as raw text</summary>
+
+- Args:
+  - `paper_id` (str)
+  - `max_chars` (int | null): truncation limit
+- Returns: markdown string (may include a `[truncated: ...]` marker)
+
+</details>
+
+<details>
+<summary><strong>get_database_stats()</strong> — snapshot-level stats</summary>
+
+- Returns:
+  - `total`
+  - `years`, `months`: list of `{ value, paper_count }`
+  - `authors`, `venues`, `institutions`, `keywords`, `tags`: top lists of `{ value, paper_count }`
+
+</details>
+
+<details>
+<summary><strong>list_top_facets(category, limit=20)</strong> — top values for one facet</summary>
+
+- Args:
+  - `category`: `author | venue | keyword | institution | tag`
+  - `limit` (int)
+- Returns: list of `{ value, paper_count }`
+
+</details>
+
+<details>
+<summary><strong>filter_papers(author=None, venue=None, year=None, keyword=None, tag=None, limit=10)</strong> — structured filtering</summary>
+
+- Args (all optional except `limit`):
+  - `author`, `venue`, `keyword`, `tag`: substring match
+  - `year`: exact match
+  - `limit` (int): number of results (clamped)
+- Returns: list of `{ paper_id, title, year, venue }`
+
+</details>
+
+#### MCP Resources (URI access)
+
+<details>
+<summary><strong>paper://{paper_id}/metadata</strong> — metadata JSON</summary>
+
+Returns the same content as `get_paper_metadata(paper_id)` (as a JSON string).
+
+</details>
+
+<details>
+<summary><strong>paper://{paper_id}/summary</strong> — preferred summary JSON</summary>
+
+Returns the same content as `get_paper_summary(paper_id)` (preferred template; JSON string).
+
+</details>
+
+<details>
+<summary><strong>paper://{paper_id}/summary/{template}</strong> — summary JSON for template</summary>
+
+Returns the same content as `get_paper_summary(paper_id, template=template)` (JSON string).
+
+</details>
+
+<details>
+<summary><strong>paper://{paper_id}/source</strong> — source markdown</summary>
+
+Returns the same content as `get_paper_source(paper_id)` (markdown string).
+
+</details>
+
+<details>
+<summary><strong>paper://{paper_id}/translation/{lang}</strong> — translated markdown</summary>
+
+Returns translated markdown for `lang` (e.g. `zh`, `ja`) when available.
+
+</details>
+
 ### 4) Frontend (static build or dev)
 
 ```bash
@@ -811,7 +959,7 @@ docker run --rm -p 8899:8899 \
 ```
 
 Notes:
-- nginx listens on 8899 and proxies `/api` to the internal API at `127.0.0.1:8000`.
+- nginx listens on 8899 and proxies `/api` and `/mcp` to the internal API at `127.0.0.1:8000`.
 - Mount your snapshot DB to `/db/papers.db` inside the container.
 - Mount snapshot static assets to `/static` when serving assets from this container (default `PAPER_DB_STATIC_BASE` is `/static`).
 - If `PAPER_DB_STATIC_BASE` is a full URL (e.g. `https://static.example.com`), nginx still serves the frontend locally, while API responses use that external static base for asset links.

{deepresearch_flow-0.7.0 → deepresearch_flow-0.7.2}/README.md

@@ -457,7 +457,14 @@ server {
   }
 
   location /api/ {
-    proxy_pass http://127.0.0.1:8001/;
+    proxy_pass http://127.0.0.1:8001;
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+  }
+
+  location ^~ /mcp {
+    proxy_pass http://127.0.0.1:8001;
     proxy_set_header Host $host;
     proxy_set_header X-Real-IP $remote_addr;
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
@@ -492,6 +499,147 @@ uv run deepresearch-flow paper db api serve \
   --host 0.0.0.0 --port 8001
 ```
 
+### 3.1) MCP (FastMCP Streamable HTTP)
+
+This project exposes an MCP server mounted at `/mcp` on the snapshot API:
+
+- Endpoint: `http://<host>:8001/mcp` (same host/port as `paper db api serve`)
+- Transport: Streamable HTTP via `POST` only (no SSE; `GET` returns 405)
+- Protocol header: optional `mcp-protocol-version` (`2025-03-26` or `2025-06-18`)
+- Static reads: summary/source/translation are served as **text content** by reading snapshot static assets (local-first via `PAPER_DB_STATIC_EXPORT_DIR`, HTTP fallback via `PAPER_DB_STATIC_BASE` / `PAPER_DB_STATIC_BASE_URL`)
+
+Optional (avoid HTTP fetch by reading exported assets directly on the API host):
+
+```bash
+export PAPER_DB_STATIC_EXPORT_DIR=/data/paper-static
+```
+
+#### MCP Tools (API functions)
+
+<details>
+<summary><strong>search_papers(query, limit=10)</strong> — full-text search (relevance-ranked)</summary>
+
+- Args:
+  - `query` (str): keywords / topic query
+  - `limit` (int): number of results (clamped to API max page size)
+- Returns: list of `{ paper_id, title, year, venue, snippet_markdown }`
+
+</details>
+
+<details>
+<summary><strong>search_papers_by_keyword(keyword, limit=10)</strong> — facet keyword search</summary>
+
+- Args:
+  - `keyword` (str): keyword substring
+  - `limit` (int): number of results (clamped)
+- Returns: list of `{ paper_id, title, year, venue, snippet_markdown }`
+
+</details>
+
+<details>
+<summary><strong>get_paper_metadata(paper_id)</strong> — metadata + available summary templates</summary>
+
+- Args:
+  - `paper_id` (str)
+- Returns: dict with:
+  - `paper_id`, `title`, `year`, `venue`
+  - `doi`, `arxiv_id`, `openreview_id`, `paper_pw_url`
+  - `preferred_summary_template`, `available_summary_templates`
+
+</details>
+
+<details>
+<summary><strong>get_paper_summary(paper_id, template=None, max_chars=None)</strong> — summary JSON as raw text</summary>
+
+- Notes:
+  - Uses `preferred_summary_template` if `template` is omitted
+  - Returns the **full JSON content** (not a URL)
+- Args:
+  - `paper_id` (str)
+  - `template` (str | null)
+  - `max_chars` (int | null): truncation limit
+- Returns: JSON string (may include a `[truncated: ...]` marker)
+
+</details>
+
+<details>
+<summary><strong>get_paper_source(paper_id, max_chars=None)</strong> — source markdown as raw text</summary>
+
+- Args:
+  - `paper_id` (str)
+  - `max_chars` (int | null): truncation limit
+- Returns: markdown string (may include a `[truncated: ...]` marker)
+
+</details>
+
+<details>
+<summary><strong>get_database_stats()</strong> — snapshot-level stats</summary>
+
+- Returns:
+  - `total`
+  - `years`, `months`: list of `{ value, paper_count }`
+  - `authors`, `venues`, `institutions`, `keywords`, `tags`: top lists of `{ value, paper_count }`
+
+</details>
+
+<details>
+<summary><strong>list_top_facets(category, limit=20)</strong> — top values for one facet</summary>
+
+- Args:
+  - `category`: `author | venue | keyword | institution | tag`
+  - `limit` (int)
+- Returns: list of `{ value, paper_count }`
+
+</details>
+
+<details>
+<summary><strong>filter_papers(author=None, venue=None, year=None, keyword=None, tag=None, limit=10)</strong> — structured filtering</summary>
+
+- Args (all optional except `limit`):
+  - `author`, `venue`, `keyword`, `tag`: substring match
+  - `year`: exact match
+  - `limit` (int): number of results (clamped)
+- Returns: list of `{ paper_id, title, year, venue }`
+
+</details>
+
+#### MCP Resources (URI access)
+
+<details>
+<summary><strong>paper://{paper_id}/metadata</strong> — metadata JSON</summary>
+
+Returns the same content as `get_paper_metadata(paper_id)` (as a JSON string).
+
+</details>
+
+<details>
+<summary><strong>paper://{paper_id}/summary</strong> — preferred summary JSON</summary>
+
+Returns the same content as `get_paper_summary(paper_id)` (preferred template; JSON string).
+
+</details>
+
+<details>
+<summary><strong>paper://{paper_id}/summary/{template}</strong> — summary JSON for template</summary>
+
+Returns the same content as `get_paper_summary(paper_id, template=template)` (JSON string).
+
+</details>
+
+<details>
+<summary><strong>paper://{paper_id}/source</strong> — source markdown</summary>
+
+Returns the same content as `get_paper_source(paper_id)` (markdown string).
+
+</details>
+
+<details>
+<summary><strong>paper://{paper_id}/translation/{lang}</strong> — translated markdown</summary>
+
+Returns translated markdown for `lang` (e.g. `zh`, `ja`) when available.
+
+</details>
+
 ### 4) Frontend (static build or dev)
 
 ```bash
@@ -748,7 +896,7 @@ docker run --rm -p 8899:8899 \
 ```
 
 Notes:
-- nginx listens on 8899 and proxies `/api` to the internal API at `127.0.0.1:8000`.
+- nginx listens on 8899 and proxies `/api` and `/mcp` to the internal API at `127.0.0.1:8000`.
 - Mount your snapshot DB to `/db/papers.db` inside the container.
 - Mount snapshot static assets to `/static` when serving assets from this container (default `PAPER_DB_STATIC_BASE` is `/static`).
 - If `PAPER_DB_STATIC_BASE` is a full URL (e.g. `https://static.example.com`), nginx still serves the frontend locally, while API responses use that external static base for asset links.

{deepresearch_flow-0.7.0 → deepresearch_flow-0.7.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "deepresearch-flow"
-version = "0.7.0"
+version = "0.7.2"
 description = "Workflow tools for paper extraction, review, and research automation."
 readme = { file = "README.md", content-type = "text/markdown" }
 license = { file = "LICENSE" }

{deepresearch_flow-0.7.0 → deepresearch_flow-0.7.2}/python/deepresearch_flow/__init__.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "0.1.0"
+__version__ = "0.7.1"

{deepresearch_flow-0.7.0 → deepresearch_flow-0.7.2}/python/deepresearch_flow/paper/snapshot/mcp_server.py

@@ -227,18 +227,24 @@ def _load_summary_json(paper_id: str, template: str | None) -> tuple[str | None,
     conn = _open_ro_conn(cfg.snapshot_db)
     try:
         row = conn.execute(
-            "SELECT preferred_summary_template, summary_asset_paths_json FROM paper WHERE paper_id = ?",
+            "SELECT preferred_summary_template FROM paper WHERE paper_id = ?",
             (paper_id,),
         ).fetchone()
         if not row:
             return None, None
-        preferred = row["preferred_summary_template"]
-        asset_paths = json.loads(row["summary_asset_paths_json"] or "{}")
-        available = sorted(asset_paths.keys())
-        selected = template if template else preferred
-        if not selected or selected not in asset_paths:
+        preferred = str(row["preferred_summary_template"] or "")
+        template_rows = conn.execute(
+            "SELECT template_tag FROM paper_summary WHERE paper_id = ?",
+            (paper_id,),
+        ).fetchall()
+        available = sorted((str(item["template_tag"]) for item in template_rows), key=str.lower)
+        selected = (template or preferred).strip()
+        if not selected or selected not in set(available):
             return None, available
-        rel_path = asset_paths[selected]
+        if template:
+            rel_path = f"summary/{paper_id}/{selected}.json"
+        else:
+            rel_path = f"summary/{paper_id}.json"
         return _load_static_text(rel_path), available
     finally:
         conn.close()
@@ -267,15 +273,12 @@ def _load_translation_markdown(paper_id: str, lang: str) -> str | None:
     conn = _open_ro_conn(cfg.snapshot_db)
     try:
         row = conn.execute(
-            "SELECT translations_json FROM paper WHERE paper_id = ?",
-            (paper_id,),
+            "SELECT md_content_hash FROM paper_translation WHERE paper_id = ? AND lang = ?",
+            (paper_id, lang),
         ).fetchone()
-        if not row or not row["translations_json"]:
-            return None
-        translations = json.loads(row["translations_json"])
-        rel_path = translations.get(lang)
-        if not rel_path:
+        if not row or not row["md_content_hash"]:
             return None
+        rel_path = f"md_translate/{lang}/{row['md_content_hash']}.md"
         return _load_static_text(rel_path)
     finally:
         conn.close()
@@ -296,22 +299,32 @@ def search_papers(query: str, limit: int = 10) -> list[dict[str, Any]]:
     
     conn = _open_ro_conn(cfg.snapshot_db)
     try:
+        match_expr = rewrite_search_query(query)
+        if not match_expr:
+            return []
         cur = conn.execute(
             """
-            SELECT paper_id, title, year, venue, abstract
-            FROM paper_search
-            WHERE paper_search MATCH ?
+            SELECT
+              p.paper_id,
+              p.title,
+              p.year,
+              p.venue,
+              snippet(paper_fts, -1, '[[[', ']]]', '…', 30) AS snippet_markdown,
+              bm25(paper_fts, 5.0, 3.0, 1.0, 1.0, 2.0) AS rank
+            FROM paper_fts
+            JOIN paper p ON p.paper_id = paper_fts.paper_id
+            WHERE paper_fts MATCH ?
             ORDER BY rank
             LIMIT ?
             """,
-            (rewrite_search_query(query), limit),
+            (match_expr, limit),
         )
         rows = cur.fetchall()
         results: list[dict[str, Any]] = []
         for row in rows:
-            snippet = str(row["abstract"] or "")
+            snippet = str(row["snippet_markdown"] or "")
             snippet = remove_cjk_spaces(snippet)
-            snippet, markers = merge_adjacent_markers(snippet)
+            snippet = merge_adjacent_markers(snippet)
             results.append({
                 "paper_id": str(row["paper_id"]),
                 "title": str(row["title"]),
@@ -337,7 +350,7 @@ def search_papers_by_keyword(keyword: str, limit: int = 10) -> list[dict[str, An
     try:
         rows = conn.execute(
             """
-            SELECT DISTINCT p.paper_id, p.title, p.year, p.venue, p.abstract
+            SELECT DISTINCT p.paper_id, p.title, p.year, p.venue, p.summary_preview
             FROM paper p
             JOIN paper_keyword pk ON pk.paper_id = p.paper_id
             JOIN keyword k ON k.keyword_id = pk.keyword_id
@@ -349,9 +362,9 @@ def search_papers_by_keyword(keyword: str, limit: int = 10) -> list[dict[str, An
         ).fetchall()
         results: list[dict[str, Any]] = []
         for row in rows:
-            snippet = str(row["abstract"] or "")
+            snippet = str(row["summary_preview"] or "")
             snippet = remove_cjk_spaces(snippet)
-            snippet, markers = merge_adjacent_markers(snippet)
+            snippet = merge_adjacent_markers(snippet)
             results.append({
                 "paper_id": str(row["paper_id"]),
                 "title": str(row["title"]),
@@ -377,26 +390,27 @@ def get_paper_metadata(paper_id: str) -> dict[str, Any]:
     try:
         row = conn.execute(
             """
-            SELECT paper_id, title, year, venue, doi, arxiv_id, openreview_id, paper_pw_url,
-                   preferred_summary_template, summary_asset_paths_json
+            SELECT paper_id, title, year, venue, preferred_summary_template
             FROM paper WHERE paper_id = ?
             """,
             (paper_id,),
         ).fetchone()
         if not row:
             raise McpToolError("not_found", "paper not found", paper_id=paper_id)
-        
-        asset_paths = json.loads(row["summary_asset_paths_json"] or "{}")
-        available = sorted(asset_paths.keys())
+        template_rows = conn.execute(
+            "SELECT template_tag FROM paper_summary WHERE paper_id = ?",
+            (paper_id,),
+        ).fetchall()
+        available = sorted((str(item["template_tag"]) for item in template_rows), key=str.lower)
         return {
             "paper_id": str(row["paper_id"]),
             "title": str(row["title"]),
             "year": str(row["year"]),
             "venue": str(row["venue"]),
-            "doi": row["doi"],
-            "arxiv_id": row["arxiv_id"],
-            "openreview_id": row["openreview_id"],
-            "paper_pw_url": row["paper_pw_url"],
+            "doi": None,
+            "arxiv_id": None,
+            "openreview_id": None,
+            "paper_pw_url": None,
             "preferred_summary_template": row["preferred_summary_template"],
             "available_summary_templates": available,
         }