deepresearch-flow 0.2.1__py3-none-any.whl → 0.3.0__py3-none-any.whl

deepresearch_flow-0.3.0.dist-info/METADATA

@@ -0,0 +1,306 @@
+Metadata-Version: 2.4
+Name: deepresearch-flow
+Version: 0.3.0
+Summary: Workflow tools for paper extraction, review, and research automation.
+Author-email: DengQi <dengqi935@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2025 DengQi
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/nerdneilsfield/ai-deepresearch-flow
+Project-URL: Repository, https://github.com/nerdneilsfield/ai-deepresearch-flow
+Project-URL: Issues, https://github.com/nerdneilsfield/ai-deepresearch-flow/issues
+Keywords: research,papers,pdf,ocr,llm,workflow
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: anthropic>=0.28.0
+Requires-Dist: click>=8.1.7
+Requires-Dist: coloredlogs>=15.0.1
+Requires-Dist: dashscope>=1.20.0
+Requires-Dist: google-auth>=2.0.0
+Requires-Dist: google-genai>=0.5.0
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: jinja2>=3.1.3
+Requires-Dist: json-repair>=0.31.0
+Requires-Dist: jsonschema>=4.21.1
+Requires-Dist: markdown-it-py>=3.0.0
+Requires-Dist: mdit-py-plugins>=0.4.0
+Requires-Dist: pypdf>=3.0.0
+Requires-Dist: pybtex>=0.24.0
+Requires-Dist: rich>=13.7.1
+Requires-Dist: rumdl>=0.0.214
+Requires-Dist: starlette>=0.37.2
+Requires-Dist: tqdm>=4.66.4
+Requires-Dist: uvicorn>=0.27.1
+Dynamic: license-file
+
+<p align="center">
+  <img src=".github/assets/logo.png" width="140" alt="ai-deepresearch-flow logo" />
+</p>
+
+<h3 align="center">ai-deepresearch-flow</h3>
+
+<p align="center">
+  <em>From documents to deep research insight — automatically.</em>
+</p>
+
+<p align="center">
+  <a href="README.md">English</a> | <a href="README_ZH.md">中文</a>
+</p>
+
+<p align="center">
+  <a href="https://github.com/nerdneilsfield/ai-deepresearch-flow/actions">
+    <img src="https://img.shields.io/github/actions/workflow/status/nerdneilsfield/ai-deepresearch-flow/push-to-pypi.yml?style=flat-square" />
+  </a>
+  <a href="https://pypi.org/project/deepresearch-flow/">
+    <img src="https://img.shields.io/pypi/v/deepresearch-flow?style=flat-square" />
+  </a>
+  <a href="https://pypi.org/project/deepresearch-flow/">
+    <img src="https://img.shields.io/pypi/pyversions/deepresearch-flow?style=flat-square" />
+  </a>
+  <a href="https://hub.docker.com/r/nerdneils/deepresearch-flow">
+    <img src="https://img.shields.io/docker/v/nerdneils/deepresearch-flow?style=flat-square" />
+  </a>
+  <a href="https://ghcr.io/nerdneilsfield/deepresearch-flow">
+    <img src="https://img.shields.io/badge/ghcr.io-nerdneilsfield%2Fdeepresearch-flow-0f172a?style=flat-square" />
+  </a>
+  <a href="https://github.com/nerdneilsfield/ai-deepresearch-flow/blob/main/LICENSE">
+    <img src="https://img.shields.io/github/license/nerdneilsfield/ai-deepresearch-flow?style=flat-square" />
+  </a>
+  <a href="https://github.com/nerdneilsfield/ai-deepresearch-flow/stargazers">
+    <img src="https://img.shields.io/github/stars/nerdneilsfield/ai-deepresearch-flow?style=flat-square" />
+  </a>
+  <a href="https://pypi.org/project/deepresearch-flow">
+  <img alt="PyPI - Version" src="https://img.shields.io/pypi/v/deepresearch-flow">
+  </a>
+  <a href="https://github.com/nerdneilsfield/ai-deepresearch-flow/issues">
+    <img src="https://img.shields.io/github/issues/nerdneilsfield/ai-deepresearch-flow?style=flat-square" />
+  </a>
+</p>
+
+---
+
+## The Core Pain Points
+
+- **OCR Chaos**: Raw markdown from OCR tools is often broken -- tables drift, formulas break, and references are non-clickable.
+- **Translation Nightmares**: Translating technical papers often destroys code blocks, LaTeX formulas, and table structures.
+- **Information Overload**: Extracting structured insights (authors, venues, summaries) from hundreds of PDFs manually is impossible.
+- **Context Switching**: Managing PDFs, summaries, and translations in different windows kills focus.
+
+## The Solution
+
+DeepResearch Flow provides a unified pipeline to **Repair**, **Translate**, **Extract**, and **Serve** your research library.
+
+## Key Features
+
+- **Smart Extraction**: Turn unstructured Markdown into schema-enforced JSON (summaries, metadata, Q&A) using LLMs (OpenAI, Claude, Gemini, etc.).
+- **Precision Translation**: Translate OCR Markdown to Chinese/Japanese (`.zh.md`, `.ja.md`) while **freezing** formulas, code, tables, and references. No more broken layout.
+- **Local Knowledge DB**: A high-performance local Web UI to browse papers with **Split View** (Source vs. Translated vs. Summary), full-text search, and multi-dimensional filtering.
+- **OCR Post-Processing**: Automatically fix broken references (`[1]` -> `[^1]`), merge split paragraphs, and standardize layouts.
+
+---
+
+## Quick Start
+
+### 1) Installation
+
+```bash
+# Recommended: using uv for speed
+uv pip install deepresearch-flow
+
+# Or standard pip
+pip install deepresearch-flow
+```
+
+### 2) Configuration
+
+Set up your LLM providers. We support OpenAI, Claude, Gemini, Ollama, and more.
+
+```bash
+cp config.example.toml config.toml
+# Edit config.toml to add your API keys (e.g., env:OPENAI_API_KEY)
+```
+
+### 3) The "Zero to Hero" Workflow
+
+#### Step 1: Extract Insights
+
+Scan a folder of markdown files and extract structured summaries.
+
+```bash
+uv run deepresearch-flow paper extract \
+  --input ./docs \
+  --model openai/gpt-4o-mini \
+  --prompt-template deep_read
+```
+
+#### Step 2: Translate Safely
+
+Translate papers to Chinese, protecting LaTeX and tables.
+
+```bash
+uv run deepresearch-flow translator translate \
+  --input ./docs \
+  --target-lang zh \
+  --model openai/gpt-4o-mini \
+  --fix-level moderate
+```
+
+#### Step 3: Serve Your Database
+
+Launch a local UI to read and manage your papers.
+
+```bash
+uv run deepresearch-flow paper db serve \
+  --input paper_infos.json \
+  --md-root ./docs \
+  --md-translated-root ./docs \
+  --host 127.0.0.1
+```
+
+---
+
+## Comprehensive Guide
+
+<details>
+<summary><strong>1. Translator: OCR-Safe Translation</strong></summary>
+
+The translator module is built for scientific documents. It uses a node-based architecture to ensure stability.
+
+- Structure Protection: automatically detects and "freezes" code blocks, LaTeX (`$$...$$`), HTML tables, and images before sending text to the LLM.
+- OCR Repair: use `--fix-level` to merge broken paragraphs and convert text references (`[1]`) to clickable Markdown footnotes (`[^1]`).
+- Context-Aware: supports retries for failed chunks and falls back gracefully.
+
+```bash
+# Translate with structure protection and OCR repairs
+uv run deepresearch-flow translator translate \
+  --input ./paper.md \
+  --target-lang ja \
+  --fix-level aggressive \
+  --model claude/claude-3-5-sonnet-20240620
+```
+
+</details>
+
+<details>
+<summary><strong>2. Paper Extract: Structured Knowledge</strong></summary>
+
+Turn loose markdown files into a queryable database.
+
+- Templates: built-in prompts like `simple`, `eight_questions`, and `deep_read` guide the LLM to extract specific insights.
+- Async and throttled: precise control over concurrency (`--max-concurrency`) and rate limits (`--sleep-every`).
+- Incremental: skips already processed files; resumes from where you left off.
+
+```bash
+uv run deepresearch-flow paper extract \
+  --input ./library \
+  --output paper_data.json \
+  --template-dir ./my-custom-prompts \
+  --max-concurrency 10
+```
+
+</details>
+
+<details>
+<summary><strong>3. Database and UI: Your Personal ArXiv</strong></summary>
+
+The db serve command creates a local research station.
+
+- Split View: read the original PDF/Markdown on the left and the Summary/Translation on the right.
+- Full Text Search: search by title, author, year, or content tags (`tag:fpga year:2023..2024`).
+- Stats: visualize publication trends and keyword frequencies.
+- PDF Viewer: built-in PDF.js viewer prevents cross-origin issues with local files.
+
+```bash
+uv run deepresearch-flow paper db serve \
+  --input paper_infos.json \
+  --pdf-root ./pdfs \
+  --cache-dir .cache/db
+```
+
+</details>
+
+<details>
+<summary><strong>4. Recognize: OCR Post-Processing</strong></summary>
+
+Tools to clean up raw outputs from OCR engines like MinerU.
+
+- Embed Images: convert local image links to Base64 for a portable single-file Markdown.
+- Unpack Images: extract Base64 images back to files.
+- Organize: flatten nested OCR output directories.
+- Fix: apply OCR fixes and rumdl formatting during organize, or as a standalone step.
+
+```bash
+uv run deepresearch-flow recognize md embed --input ./raw_ocr --output ./clean_md
+```
+
+```bash
+# Organize MinerU output and apply OCR fixes
+uv run deepresearch-flow recognize organize \
+  --input ./mineru_outputs \
+  --output-simple ./ocr_md \
+  --fix
+
+# Fix and format existing markdown outputs
+uv run deepresearch-flow recognize fix \
+  --input ./ocr_md \
+  --output ./ocr_md_fixed
+
+# Fix in place
+uv run deepresearch-flow recognize fix \
+  --input ./ocr_md \
+  --in-place
+```
+
+</details>
+
+---
+
+## Docker Support
+
+Don't want to manage Python environments?
+
+```bash
+docker run --rm -v $(pwd):/app -it ghcr.io/nerdneilsfield/deepresearch-flow --help
+```
+
+## Configuration
+
+The config.toml is your control center. It supports:
+
+- Multiple Providers: mix and match OpenAI, DeepSeek (DashScope), Gemini, Claude, and Ollama.
+- Model Routing: explicit routing to specific models (`--model provider/model_name`).
+- Environment Variables: keep secrets safe using `env:VAR_NAME` syntax.
+
+See `config.example.toml` for a full reference.
+
+---
+
+<p align="center">
+  Built with love for the Open Science community.
+</p>

{deepresearch_flow-0.2.1.dist-info → deepresearch_flow-0.3.0.dist-info}/RECORD

@@ -1,12 +1,12 @@
 deepresearch_flow/__init__.py,sha256=rjP9ES4zJCfEN_MCDYAYPL1mNJZGjojdmbRwnZ9FlEk,83
 deepresearch_flow/__main__.py,sha256=Ceo0rMTOhHhwFPD-HyDDagenNsmWEzPmsdYLI7kwKVA,115
-deepresearch_flow/cli.py,sha256=WhPhs-Cg4kHow0h0KTVaGTjQVXCPrlNvMyvgxCD8qgI,371
+deepresearch_flow/cli.py,sha256=t4oowCNWldL0DrVJ4d0UlRkuGU2qHej_G0mAc_quteQ,455
 deepresearch_flow/paper/__init__.py,sha256=sunaOkcgAJBrfmcaJTumcWbPGVUSGWvOv2a2Yidzy0A,43
 deepresearch_flow/paper/cli.py,sha256=4UY3KHi6BUGztL1vB4w0cCMiIAo9KNxrfQn1GBHt6fA,11153
-deepresearch_flow/paper/config.py,sha256=5uGTWfAfzpv4w_JxC0w6GF2teaxF5b3rD8LaDqPVshU,8611
-deepresearch_flow/paper/db.py,sha256=uX-gblqh-ltoMO6mv0KPAm-sgNaRz46jaN0kxtzvP8s,33242
+deepresearch_flow/paper/config.py,sha256=totVBGzouh0KS6mhRNPneXZYPuuw0SHiOGdO3r6HSfc,9289
+deepresearch_flow/paper/db.py,sha256=ymVLzSEXDksdhLNSdvNA2IWLzT5lQOG1CpJlPU9CSQ8,33586
 deepresearch_flow/paper/extract.py,sha256=ID1dd2r6LTB0kRF4qBSH6bGtBGv0znw--g_mXYBcoeU,32314
-deepresearch_flow/paper/llm.py,sha256=R4rmFoYnGq_JiQODr4Jzk5j8U-j2NSYUXex6eR-WHXg,3929
+deepresearch_flow/paper/llm.py,sha256=mHfs5IkT3Q6BOh46MDlfUmgVTX24WRf0IKKoOnN8nV8,4007
 deepresearch_flow/paper/prompts.py,sha256=mV7cEXw8pwukBUE4Trah0SjEPSSDgg5-RGaNaUdo4EU,519
 deepresearch_flow/paper/render.py,sha256=KeccrRGf1_sxoaiT6SUDkFRj9sStReoEwNvlw1ir7qw,2181
 deepresearch_flow/paper/schema.py,sha256=tQEVbj4R8NqNGBW6VYwW-xf5QJgV9qthrbZB-EmZTKA,1931
@@ -40,7 +40,7 @@ deepresearch_flow/paper/templates/default_paper.md.j2,sha256=3azu48534QtLtHrCwI1
 deepresearch_flow/paper/templates/eight_questions.md.j2,sha256=Ecz4CD3nd7jZ4Dg8himZkTwF4WDkk0ILWk8V728uOPI,3038
 deepresearch_flow/paper/templates/three_pass.md.j2,sha256=ZRj-NkpZePnqp0gSE8OT1dN5Lr5RW4vdOYdeVejYJW0,1576
 deepresearch_flow/paper/web/__init__.py,sha256=eQBtBjvOYsNEdivHTI0aO286SCG2c86xI02tf-0jz5I,39
-deepresearch_flow/paper/web/app.py,sha256=OB0iHU5pa7zJmP4IQAHPg4S-ucfcWBRBfHfaSDNJDTE,118325
+deepresearch_flow/paper/web/app.py,sha256=nb4uzsDJ2R5dz_WA69NKwTgVgMqAyZv5OZ88GxFTWLQ,133311
 deepresearch_flow/paper/web/query.py,sha256=vTegfm5zGVkYCd6_K3yNrXJEmKMccUUFKG9DePPcKMw,1938
 deepresearch_flow/paper/web/pdfjs/LICENSE,sha256=DVQuDIgE45qn836wDaWnYhSdxoLXgpRRKH4RuTjpRZQ,10174
 deepresearch_flow/paper/web/pdfjs/build/pdf.js,sha256=2Ddm8gpMMfvOWinZh4nN--94GxR0QdpFvh0Qeejg-Bw,568294
@@ -413,12 +413,21 @@ deepresearch_flow/paper/web/pdfjs/web/standard_fonts/LiberationSans-BoldItalic.t
 deepresearch_flow/paper/web/pdfjs/web/standard_fonts/LiberationSans-Italic.ttf,sha256=gytEBtvvI2KIANOqrSEEhTSshNfjrZVb6DuBcu2O9RI,162036
 deepresearch_flow/paper/web/pdfjs/web/standard_fonts/LiberationSans-Regular.ttf,sha256=-Kzh-JKyvZ3BeSun8Jf6dYj4T-1IMhSA4E3lOQgoIh8,139512
 deepresearch_flow/recognize/__init__.py,sha256=yMAqbdCzpdRSiwFhq9j7yx9ZWxqz_Zq3vfYlTLFCWek,33
-deepresearch_flow/recognize/cli.py,sha256=zhJi6f0Ha6UvX-Q4mdPdM9uz0SoBuCEnRwzDslMN2Eg,16276
+deepresearch_flow/recognize/cli.py,sha256=zWUsqvou2h6c5zR_myGaySvK6cG9ItJp9cJFtqqJk7Y,21597
 deepresearch_flow/recognize/markdown.py,sha256=y-PMJbGqrfWCNBVGanXK1M4OuMP9e1eqh7HDYye5a7Q,8757
-deepresearch_flow/recognize/organize.py,sha256=GSLmo037rpARSecaPxNCuIlLBbbilx8msWFJDqYJ4hc,3561
-deepresearch_flow-0.2.1.dist-info/licenses/LICENSE,sha256=hT8F2Py1pe6flxq3Ufdm2UKFk0B8CBm0aAQfsLXfvjw,1063
-deepresearch_flow-0.2.1.dist-info/METADATA,sha256=n7JgsLFBDM80UkhTHCtKn2biC-_mAA-sGBilxnIPzso,15250
-deepresearch_flow-0.2.1.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-deepresearch_flow-0.2.1.dist-info/entry_points.txt,sha256=1uIKscs0YRMg_mFsg9NjsaTt4CvQqQ_-zGERUKhhL_Y,65
-deepresearch_flow-0.2.1.dist-info/top_level.txt,sha256=qBl4RvPJNJUbL8CFfMNWxY0HpQLx5RlF_ko-z_aKpm0,18
-deepresearch_flow-0.2.1.dist-info/RECORD,,
+deepresearch_flow/recognize/organize.py,sha256=-KVzuwNjiT2bLwqwLwcguEMQYxnGiZXjLNlov_oXSTo,5237
+deepresearch_flow/translator/__init__.py,sha256=iaAkufvEELVKNbcs08Nh7bkTO4JlkT3rT_JIBP9jGfc,26
+deepresearch_flow/translator/cli.py,sha256=BceOZhQuN9s5kqhpvLJuwpbB5J0MY1ucWUKw0jXWUPc,16872
+deepresearch_flow/translator/config.py,sha256=0JI4VBLIzT039YscfEb5hqtCWCu8P2bJIgnAfIAhFmU,502
+deepresearch_flow/translator/engine.py,sha256=dLKKUjmptkLXhIs5ZsIUonmKI9bS8Se4tOnp7fADIYU,36800
+deepresearch_flow/translator/fixers.py,sha256=6lRfeic8aGhsFDAWVb2p-QCXPzNkNNWMWXbQev-qyTw,15199
+deepresearch_flow/translator/placeholder.py,sha256=mEgqA-dPdOsIhno0h_hzfpXpY2asb4A7UQEYV3tcnP8,2097
+deepresearch_flow/translator/prompts.py,sha256=kl_9O2YvmtXC1w6WLnsLuVZKz4mcOtUF887SiTaOvc0,4754
+deepresearch_flow/translator/protector.py,sha256=sXwNJ1Y8tyPm7dgm8-7S8HkcPe23TGsBdwRxH6mKL70,11291
+deepresearch_flow/translator/segment.py,sha256=rBFMCLTrvm2GrPc_hNFymi-8Ih2DAtUQlZHCRE9nLaM,5146
+deepresearch_flow-0.3.0.dist-info/licenses/LICENSE,sha256=hT8F2Py1pe6flxq3Ufdm2UKFk0B8CBm0aAQfsLXfvjw,1063
+deepresearch_flow-0.3.0.dist-info/METADATA,sha256=AJ4RfKW-V9BPhrrlFSP8stAoXG4SwpF-AvZH5HEtWyw,10831
+deepresearch_flow-0.3.0.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+deepresearch_flow-0.3.0.dist-info/entry_points.txt,sha256=1uIKscs0YRMg_mFsg9NjsaTt4CvQqQ_-zGERUKhhL_Y,65
+deepresearch_flow-0.3.0.dist-info/top_level.txt,sha256=qBl4RvPJNJUbL8CFfMNWxY0HpQLx5RlF_ko-z_aKpm0,18
+deepresearch_flow-0.3.0.dist-info/RECORD,,