deepresearch-flow 0.5.0__tar.gz → 0.6.0__tar.gz

{deepresearch_flow-0.5.0/python/deepresearch_flow.egg-info → deepresearch_flow-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: deepresearch-flow
-Version: 0.5.0
+Version: 0.6.0
 Summary: Workflow tools for paper extraction, review, and research automation.
 Author-email: DengQi <dengqi935@gmail.com>
 License: MIT License
@@ -38,24 +38,24 @@ Classifier: Topic :: Scientific/Engineering :: Information Analysis
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: anthropic>=0.28.0
+Requires-Dist: anthropic>=0.77.0
 Requires-Dist: click>=8.1.7
 Requires-Dist: coloredlogs>=15.0.1
-Requires-Dist: dashscope>=1.20.0
-Requires-Dist: google-auth>=2.0.0
-Requires-Dist: google-genai>=0.5.0
+Requires-Dist: dashscope>=1.25.10
+Requires-Dist: google-auth>=2.48.0
+Requires-Dist: google-genai>=1.60.0
 Requires-Dist: httpx>=0.27.0
 Requires-Dist: jinja2>=3.1.3
-Requires-Dist: json-repair>=0.31.0
-Requires-Dist: jsonschema>=4.21.1
+Requires-Dist: json-repair>=0.55.1
+Requires-Dist: jsonschema>=4.26.0
 Requires-Dist: markdown-it-py>=3.0.0
 Requires-Dist: mdit-py-plugins>=0.4.0
-Requires-Dist: pypdf>=3.0.0
+Requires-Dist: pypdf>=6.6.2
 Requires-Dist: pylatexenc>=2.10
 Requires-Dist: pybtex>=0.24.0
-Requires-Dist: rich>=13.7.1
-Requires-Dist: rumdl>=0.0.218
-Requires-Dist: starlette>=0.37.2
+Requires-Dist: rich>=14.3.1
+Requires-Dist: rumdl>=0.1.6
+Requires-Dist: starlette>=0.52.1
 Requires-Dist: tqdm>=4.66.4
 Requires-Dist: uvicorn>=0.27.1
 Dynamic: license-file
@@ -122,7 +122,9 @@ DeepResearch Flow provides a unified pipeline to **Repair**, **Translate**, **Ex
 - **Smart Extraction**: Turn unstructured Markdown into schema-enforced JSON (summaries, metadata, Q&A) using LLMs (OpenAI, Claude, Gemini, etc.).
 - **Precision Translation**: Translate OCR Markdown to Chinese/Japanese (`.zh.md`, `.ja.md`) while **freezing** formulas, code, tables, and references. No more broken layout.
 - **Local Knowledge DB**: A high-performance local Web UI to browse papers with **Split View** (Source vs. Translated vs. Summary), full-text search, and multi-dimensional filtering.
+- **Snapshot + API Serve**: Build a production-ready SQLite snapshot with static assets, then serve a read-only JSON API for a separate frontend.
 - **Coverage Compare**: Compare JSON/PDF/Markdown/Translated datasets to find missing artifacts and export CSV reports.
+- **Matched Export**: Extract matched JSON or translated Markdown after coverage checks.
 - **OCR Post-Processing**: Automatically fix broken references (`[1]` -> `[^1]`), merge split paragraphs, and standardize layouts.
 
 ---
@@ -148,6 +150,16 @@ cp config.example.toml config.toml
 # Edit config.toml to add your API keys (e.g., env:OPENAI_API_KEY)
 ```
 
+Multiple keys per provider are supported. Keys rotate per request and enter a short cooldown on retryable errors.
+You can also provide quota metadata per key:
+
+```toml
+api_keys = [
+  "env:OPENAI_API_KEY",
+  { key = "env:OPENAI_API_KEY_2", quota_duration = 18000, reset_time = "2026-01-23 18:04:25 +0800 CST", quota_error_tokens = ["exceed", "quota"] }
+]
+```
+
 ### 3) The "Zero to Hero" Workflow
 
 #### Step 1: Extract Insights
@@ -161,6 +173,31 @@ uv run deepresearch-flow paper extract \
   --prompt-template deep_read
 ```
 
+<p align="center">
+  <img src=".github/assets/extract.png" width="70%" alt="extract" />
+</p>
+
+#### Step 1.1: Verify & Retry Missing Fields
+
+Validate extracted JSON against the template schema and retry only the missing items.
+
+```bash
+uv run deepresearch-flow paper db verify \
+  --input-json ./paper_infos.json \
+  --prompt-template deep_read \
+  --output-json ./paper_verify.json
+
+uv run deepresearch-flow paper extract \
+  --input ./docs \
+  --model openai/gpt-4o-mini \
+  --prompt-template deep_read \
+  --retry-list-json ./paper_verify.json
+```
+
+<p align="center">
+  <img src=".github/assets/verify.png" width="70%" alt="verify" />
+</p>
+
 #### Step 2: Translate Safely
 
 Translate papers to Chinese, protecting LaTeX and tables.
@@ -182,20 +219,56 @@ Recommended sequence to stabilize markdown before serving:
 uv run deepresearch-flow recognize fix \
   --input ./docs \
   --in-place
+```
 
+<p align="center">
+  <img src=".github/assets/fix.png" width="70%" alt="fix" />
+</p>
+
+```bash
 # 2) Fix LaTeX formulas
 uv run deepresearch-flow recognize fix-math \
   --input ./docs \
   --model openai/gpt-4o-mini \
   --in-place
+```
+
+<p align="center">
+  <img src=".github/assets/fix-math.png" width="70%" alt="fix math" />
+</p>
 
+```bash
 # 3) Fix Mermaid diagrams
 uv run deepresearch-flow recognize fix-mermaid \
   --input ./paper_outputs \
   --json \
   --model openai/gpt-4o-mini \
   --in-place
+```
 
+<p align="center">
+  <img src=".github/assets/fix-mermaid.png" width="70%" alt="fix mermaid" />
+</p>
+
+```bash
+# (optional) Retry failed formulas/diagrams only
+uv run deepresearch-flow recognize fix-math \
+  --input ./docs \
+  --model openai/gpt-4o-mini \
+  --retry-failed
+
+uv run deepresearch-flow recognize fix-mermaid \
+  --input ./paper_outputs \
+  --json \
+  --model openai/gpt-4o-mini \
+  --retry-failed
+```
+
+<p align="center">
+  <img src=".github/assets/fix-retry-failed.png" width="70%" alt="fix retry failed" />
+</p>
+
+```bash
 # 4) Fix again to normalize formatting
 uv run deepresearch-flow recognize fix \
   --input ./docs \
@@ -214,30 +287,166 @@ uv run deepresearch-flow paper db serve \
   --host 127.0.0.1
 ```
 
+#### Step 4.5: Build Snapshot + Serve API + Frontend (Recommended)
+
+Build a production snapshot (SQLite + static assets), serve a read-only API, and run the frontend.
+
+```bash
+# 1) Build snapshot + static export
+uv run deepresearch-flow paper db snapshot build \
+  --input ./paper_infos.json \
+  --bibtex ./papers.bib \
+  --md-root ./docs \
+  --md-translated-root ./docs \
+  --pdf-root ./pdfs \
+  --output-db ./dist/paper_snapshot.db \
+  --static-export-dir ./dist/paper-static
+
+# 2) Serve static assets (CORS required for ZIP export)
+npx http-server ./dist/paper-static -p 8002 --cors
+
+# 3) Serve API (read-only)
+PAPER_DB_STATIC_BASE_URL=http://127.0.0.1:8002 \
+uv run deepresearch-flow paper db api serve \
+  --snapshot-db ./dist/paper_snapshot.db \
+  --cors-origin http://127.0.0.1:5173 \
+  --host 127.0.0.1 --port 8001
+
+# 4) Run frontend
+cd frontend
+npm install
+VITE_PAPER_DB_API_BASE=http://127.0.0.1:8001/api/v1 \
+VITE_PAPER_DB_STATIC_BASE=http://127.0.0.1:8002 \
+npm run dev
+```
+
 ---
 
+## Incremental PDF Library Workflow
+
+This workflow keeps a growing PDF library in sync without reprocessing everything.
+
+```bash
+# 1) Compare processed JSON vs new PDF library to find missing PDFs
+uv run deepresearch-flow paper db compare \
+  --input-a ./paper_infos.json \
+  --pdf-root-b ./pdfs_new \
+  --output-only-in-b ./pdfs_todo.txt
+
+# 2) Stage the missing PDFs for OCR
+uv run deepresearch-flow paper db transfer-pdfs \
+  --input-list ./pdfs_todo.txt \
+  --output-dir ./pdfs_todo \
+  --copy
+
+# (optional) use --move instead of --copy
+# uv run deepresearch-flow paper db transfer-pdfs --input-list ./pdfs_todo.txt --output-dir ./pdfs_todo --move
+
+# 3) OCR the missing PDFs (use your OCR tool; write markdowns to ./md_todo)
+
+# 4) Export matched existing assets against the new PDF library
+uv run deepresearch-flow paper db extract \
+  --input-json ./paper_infos.json \
+  --pdf-root ./pdfs_new \
+  --output-json ./paper_infos_matched.json
+
+uv run deepresearch-flow paper db extract \
+  --md-source-root ./mds \
+  --output-md-root ./mds_matched \
+  --pdf-root ./pdfs_new
+
+uv run deepresearch-flow paper db extract \
+  --md-translated-root ./translated \
+  --output-md-translated-root ./translated_matched \
+  --pdf-root ./pdfs_new \
+  --lang zh
+
+# 5) Translate + extract summaries for the new OCR markdowns
+uv run deepresearch-flow translator translate \
+  --input ./md_todo \
+  --target-lang zh \
+  --model openai/gpt-4o-mini
+
+uv run deepresearch-flow paper extract \
+  --input ./md_todo \
+  --model openai/gpt-4o-mini
+
+# 6) Merge and serve the new library (multi-input)
+uv run deepresearch-flow paper db serve \
+  --input ./paper_infos_matched.json \
+  --input ./paper_infos_new.json \
+  --md-root ./mds_matched \
+  --md-root ./md_todo \
+  --md-translated-root ./translated_matched \
+  --md-translated-root ./md_todo \
+  --pdf-root ./pdfs_new
+```
+
+## Merge Paper JSONs
+
+```bash
+# Merge multiple libraries using the same template
+uv run deepresearch-flow paper db merge library \
+  --inputs ./paper_infos_a.json \
+  --inputs ./paper_infos_b.json \
+  --output ./paper_infos_merged.json
+
+# Merge multiple templates from the same library (first input wins on shared fields)
+uv run deepresearch-flow paper db merge templates \
+  --inputs ./simple.json \
+  --inputs ./deep_read.json \
+  --output ./paper_infos_templates.json
+```
+
+Note: `paper db merge` is now split into `merge library` and `merge templates`.
+
+### Recommended: Merge templates then filter by BibTeX
+
+```bash
+# 1) Merge templates for the same library
+uv run deepresearch-flow paper db merge templates \
+  --inputs ./deep_read.json \
+  --inputs ./simple.json \
+  --output ./all.json
+
+# 2) Filter the merged set with BibTeX
+uv run deepresearch-flow paper db extract \
+  --input-bibtex ./library.bib \
+  --json ./all.json \
+  --output-json ./library_filtered.json \
+  --output-csv ./library_filtered.csv
+```
+
 ## Deployment (Static CDN)
 
-Use a separate static server (CDN) for PDFs/Markdown/images and keep the API/UI on another host.
+The recommended production setup is **front/back separation**:
+
+- **Static CDN** hosts PDFs/Markdown/images/summaries.
+- **API server** serves a read-only snapshot DB.
+- **Frontend** is a separate static app (Vite build or any static host).
+
+<p align="center">
+  <img src=".github/assets/frontend.png" width="80%" alt="frontend" />
+</p>
 
-### 1) Export static assets
+### 1) Build snapshot + static export
 
 ```bash
-uv run deepresearch-flow paper db serve \
-  --input paper_infos.json \
+uv run deepresearch-flow paper db snapshot build \
+  --input ./paper_infos.json \
+  --bibtex ./papers.bib \
   --md-root ./docs \
   --md-translated-root ./docs \
   --pdf-root ./pdfs \
-  --static-mode prod \
-  --static-base-url https://static.example.com \
+  --output-db ./dist/paper_snapshot.db \
   --static-export-dir /data/paper-static
 ```
 
 Notes:
-- The API host must be able to read the original PDF/Markdown roots to build the index and hashes.
-- The CDN host only needs the exported directory (e.g. `/data/paper-static`).
+- The build host must be able to read the original PDF/Markdown roots.
+- The CDN only needs the exported directory (e.g. `/data/paper-static`).
 
-### 2) Serve the export directory with CORS + cache headers (Caddy example)
+### 2) Serve static assets with CORS + cache headers (Caddy example)
 
 ```caddyfile
 :8002 {
@@ -259,19 +468,72 @@ Notes:
 }
 ```
 
-### 3) Start the API/UI with static base
+### 2.1) Nginx example (API + frontend on one domain, static on another)
+
+```nginx
+# Frontend + API (same domain)
+server {
+  listen 80;
+  server_name frontend.example.com;
+
+  root /var/www/paper-frontend;
+  index index.html;
+
+  location / {
+    try_files $uri /index.html;
+  }
+
+  location /api/ {
+    proxy_pass http://127.0.0.1:8001/;
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+  }
+}
+
+# Static assets (separate domain)
+server {
+  listen 80;
+  server_name static.example.com;
+
+  root /data/paper-static;
+
+  location / {
+    add_header Access-Control-Allow-Origin *;
+    add_header Access-Control-Allow-Methods "GET,HEAD,OPTIONS";
+    add_header Access-Control-Allow-Headers "*";
+    add_header Cache-Control "public, max-age=31536000, immutable";
+    try_files $uri =404;
+  }
+}
+```
+
+### 3) Start the API server (read-only)
 
 ```bash
 export PAPER_DB_STATIC_BASE_URL="https://static.example.com"
-export PAPER_DB_STATIC_MODE="prod"
-export PAPER_DB_STATIC_EXPORT_DIR="/data/paper-static"
-export PAPER_DB_PDFJS_CDN_BASE_URL="https://cdnjs.cloudflare.com/ajax/libs/pdf.js/4.0.379"
 
-uv run deepresearch-flow paper db serve \
-  --input paper_infos.json \
-  --md-root ./docs \
-  --md-translated-root ./docs \
-  --pdf-root ./pdfs
+uv run deepresearch-flow paper db api serve \
+  --snapshot-db /data/paper_snapshot.db \
+  --cors-origin https://frontend.example.com \
+  --host 0.0.0.0 --port 8001
+```
+
+### 4) Frontend (static build or dev)
+
+```bash
+cd frontend
+npm install
+
+# Dev
+VITE_PAPER_DB_API_BASE=https://api.example.com/api/v1 \
+VITE_PAPER_DB_STATIC_BASE=https://static.example.com \
+npm run dev
+
+# Build for static hosting
+VITE_PAPER_DB_API_BASE=https://api.example.com/api/v1 \
+VITE_PAPER_DB_STATIC_BASE=https://static.example.com \
+npm run build
 ```
 
 ---
@@ -286,6 +548,7 @@ The translator module is built for scientific documents. It uses a node-based ar
 - Structure Protection: automatically detects and "freezes" code blocks, LaTeX (`$$...$$`), HTML tables, and images before sending text to the LLM.
 - OCR Repair: use `--fix-level` to merge broken paragraphs and convert text references (`[1]`) to clickable Markdown footnotes (`[^1]`).
 - Context-Aware: supports retries for failed chunks and falls back gracefully.
+- Group Concurrency: use `--group-concurrency` to run multiple translation groups in parallel per document.
 
 ```bash
 # Translate with structure protection and OCR repairs
@@ -293,6 +556,7 @@ uv run deepresearch-flow translator translate \
   --input ./paper.md \
   --target-lang ja \
   --fix-level aggressive \
+  --group-concurrency 4 \
   --model claude/claude-3-5-sonnet-20240620
 ```
 
@@ -304,15 +568,61 @@ uv run deepresearch-flow translator translate \
 Turn loose markdown files into a queryable database.
 
 - Templates: built-in prompts like `simple`, `eight_questions`, and `deep_read` guide the LLM to extract specific insights.
-- Async and throttled: precise control over concurrency (`--max-concurrency`) and rate limits (`--sleep-every`).
+- Async and throttled: precise control over concurrency (`--max-concurrency`), rate limits (`--sleep-every`), and request timeout (`--timeout`).
 - Incremental: skips already processed files; resumes from where you left off.
+- Stage resume: multi-stage templates persist per-module outputs; use `--force-stage <name>` to rerun a module.
+- Stage DAG: enable `--stage-dag` (or `extract.stage_dag = true`) for dependency-aware parallelism; DAG mode only passes dependency outputs to a stage and `--dry-run` prints the per-stage plan.
+- Diagram hints: `deep_read` can emit inferred diagrams labeled `[Inferred]`; use `recognize fix-mermaid` on rendered markdown if needed.
+- Stage focus: multi-stage runs emphasize the active module and summarize others to reduce context overload.
+- Range filter: use `--start-idx/--end-idx` to slice inputs; range applies before `--retry-failed`/`--retry-failed-stages` (`--end-idx -1` = last item).
+- Retry failed stages: use `--retry-failed-stages` to re-run only failed stages (multi-stage templates); missing stages are forced to run. Retry runs keep existing results and only update retried items.
 
 ```bash
 uv run deepresearch-flow paper extract \
   --input ./library \
   --output paper_data.json \
   --template-dir ./my-custom-prompts \
-  --max-concurrency 10
+  --max-concurrency 10 \
+  --timeout 180
+
+# Extract items 0..99, then retry only failed ones from that range
+uv run deepresearch-flow paper extract \
+  --input ./library \
+  --start-idx 0 \
+  --end-idx 100 \
+  --retry-failed \
+  --model claude/claude-3-5-sonnet-20240620
+
+# Retry only failed stages in multi-stage templates
+uv run deepresearch-flow paper extract \
+  --input ./library \
+  --retry-failed-stages \
+  --model claude/claude-3-5-sonnet-20240620
+```
+
+</details>
+
+<details>
+<summary><strong>4. Recognize Fix: Repair Math and Mermaid</strong></summary>
+
+Fix broken LaTeX formulas and Mermaid diagrams in markdown or JSON outputs.
+
+- Retry Failed: use `--retry-failed` with the prior `--report` output to reprocess only failed formulas/diagrams.
+
+```bash
+uv run deepresearch-flow recognize fix-math \
+  --input ./docs \
+  --in-place \
+  --model claude/claude-3-5-sonnet-20240620 \
+  --report ./fix-math-errors.json \
+  --retry-failed
+
+uv run deepresearch-flow recognize fix-mermaid \
+  --input ./docs \
+  --in-place \
+  --model claude/claude-3-5-sonnet-20240620 \
+  --report ./fix-mermaid-errors.json \
+  --retry-failed
 ```
 
 </details>
@@ -357,7 +667,39 @@ uv run deepresearch-flow paper db compare \
 </details>
 
 <details>
-<summary><strong>5. Recognize: OCR Post-Processing</strong></summary>
+<summary><strong>5. Paper DB Extract: Matched Export</strong></summary>
+
+Extract matched JSON entries or translated Markdown after coverage comparison.
+
+```bash
+uv run deepresearch-flow paper db extract \
+  --json ./processed.json \
+  --input-bibtex ./refs.bib \
+  --pdf-root ./pdfs \
+  --output-json ./matched.json \
+  --output-csv ./extract.csv
+
+# Use a JSON reference list to filter the target JSON
+uv run deepresearch-flow paper db extract \
+  --json ./processed.json \
+  --input-json ./reference.json \
+  --pdf-root ./pdfs \
+  --output-json ./matched.json \
+  --output-csv ./extract.csv
+
+# Extract translated markdowns by language
+uv run deepresearch-flow paper db extract \
+  --md-root ./md_root \
+  --md-translated-root ./translated \
+  --lang zh \
+  --output-md-translated-root ./translated_matched \
+  --output-csv ./extract.csv
+```
+
+</details>
+
+<details>
+<summary><strong>6. Recognize: OCR Post-Processing</strong></summary>
 
 Tools to clean up raw outputs from OCR engines like MinerU.
 
@@ -420,7 +762,39 @@ uv run deepresearch-flow recognize fix-mermaid \
 Don't want to manage Python environments?
 
 ```bash
-docker run --rm -v $(pwd):/app -it ghcr.io/nerdneilsfield/deepresearch-flow --help
+docker run --rm -v $(pwd):/app -it ghcr.io/nerdneilsfield/deepresearch-flow:latest --help
+```
+
+Deploy image (API + frontend via nginx):
+
+```bash
+docker run --rm -p 8899:8899 \
+  -v $(pwd)/paper_snapshot.db:/db/papers.db \
+  -v $(pwd)/paper-static:/static \
+  ghcr.io/nerdneilsfield/deepresearch-flow:deploy-latest
+```
+
+Notes:
+- nginx listens on 8899 and proxies `/api` to the internal API at `127.0.0.1:8000`.
+- Mount your snapshot DB to `/db/papers.db` inside the container.
+- Mount snapshot static assets to `/static` when serving assets from this container (default `PAPER_DB_STATIC_BASE` is `/static`).
+- If `PAPER_DB_STATIC_BASE` is a full URL (e.g. `https://static.example.com`), nginx still serves the frontend locally, while API responses use that external static base for asset links.
+
+Docker Compose example (two modes):
+
+```bash
+docker compose -f scripts/docker/docker-compose.example.yml --profile local-static up
+# or
+docker compose -f scripts/docker/docker-compose.example.yml --profile external-static up
+```
+
+External static assets example:
+
+```bash
+docker run --rm -p 8899:8899 \
+  -v $(pwd)/paper_snapshot.db:/db/papers.db \
+  -e PAPER_DB_STATIC_BASE=https://static.example.com \
+  ghcr.io/nerdneilsfield/deepresearch-flow:deploy-latest
 ```
 
 ## Configuration