deepresearch-flow 0.3.0__tar.gz → 0.4.1__tar.gz

{deepresearch_flow-0.3.0 → deepresearch_flow-0.4.1}/MANIFEST.in

@@ -5,3 +5,5 @@ recursive-include python/deepresearch_flow/paper/templates *.j2
 recursive-include python/deepresearch_flow/paper/prompt_templates *.j2
 recursive-include python/deepresearch_flow/paper/examples *.toml
 recursive-include python/deepresearch_flow/paper/web/pdfjs *
+recursive-include python/deepresearch_flow/paper/web/templates *
+recursive-include python/deepresearch_flow/paper/web/static *

{deepresearch_flow-0.3.0/python/deepresearch_flow.egg-info → deepresearch_flow-0.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: deepresearch-flow
-Version: 0.3.0
+Version: 0.4.1
 Summary: Workflow tools for paper extraction, review, and research automation.
 Author-email: DengQi <dengqi935@gmail.com>
 License: MIT License
@@ -51,9 +51,10 @@ Requires-Dist: jsonschema>=4.21.1
 Requires-Dist: markdown-it-py>=3.0.0
 Requires-Dist: mdit-py-plugins>=0.4.0
 Requires-Dist: pypdf>=3.0.0
+Requires-Dist: pylatexenc>=2.10
 Requires-Dist: pybtex>=0.24.0
 Requires-Dist: rich>=13.7.1
-Requires-Dist: rumdl>=0.0.214
+Requires-Dist: rumdl>=0.0.218
 Requires-Dist: starlette>=0.37.2
 Requires-Dist: tqdm>=4.66.4
 Requires-Dist: uvicorn>=0.27.1
@@ -121,6 +122,7 @@ DeepResearch Flow provides a unified pipeline to **Repair**, **Translate**, **Ex
 - **Smart Extraction**: Turn unstructured Markdown into schema-enforced JSON (summaries, metadata, Q&A) using LLMs (OpenAI, Claude, Gemini, etc.).
 - **Precision Translation**: Translate OCR Markdown to Chinese/Japanese (`.zh.md`, `.ja.md`) while **freezing** formulas, code, tables, and references. No more broken layout.
 - **Local Knowledge DB**: A high-performance local Web UI to browse papers with **Split View** (Source vs. Translated vs. Summary), full-text search, and multi-dimensional filtering.
+- **Coverage Compare**: Compare JSON/PDF/Markdown/Translated datasets to find missing artifacts and export CSV reports.
 - **OCR Post-Processing**: Automatically fix broken references (`[1]` -> `[^1]`), merge split paragraphs, and standardize layouts.
 
 ---
@@ -171,7 +173,36 @@ uv run deepresearch-flow translator translate \
   --fix-level moderate
 ```
 
-#### Step 3: Serve Your Database
+#### Step 3: Repair OCR Outputs (Recommended)
+
+Recommended sequence to stabilize markdown before serving:
+
+```bash
+# 1) Fix OCR markdown (auto-detects JSON if inputs are .json)
+uv run deepresearch-flow recognize fix \
+  --input ./docs \
+  --in-place
+
+# 2) Fix LaTeX formulas
+uv run deepresearch-flow recognize fix-math \
+  --input ./docs \
+  --model openai/gpt-4o-mini \
+  --in-place
+
+# 3) Fix Mermaid diagrams
+uv run deepresearch-flow recognize fix-mermaid \
+  --input ./paper_outputs \
+  --json \
+  --model openai/gpt-4o-mini \
+  --in-place
+
+# 4) Fix again to normalize formatting
+uv run deepresearch-flow recognize fix \
+  --input ./docs \
+  --in-place
+```
+
+#### Step 4: Serve Your Database
 
 Launch a local UI to read and manage your papers.
 
@@ -246,7 +277,27 @@ uv run deepresearch-flow paper db serve \
 </details>
 
 <details>
-<summary><strong>4. Recognize: OCR Post-Processing</strong></summary>
+<summary><strong>4. Paper DB Compare: Coverage Audit</strong></summary>
+
+Compare two datasets (A/B) to find missing PDFs, markdowns, translations, or JSON items, with match metadata.
+
+```bash
+uv run deepresearch-flow paper db compare \
+  --input-a ./a.json \
+  --md-root-b ./md_root \
+  --output-csv ./compare.csv
+
+# Compare translated markdowns by language
+uv run deepresearch-flow paper db compare \
+  --md-translated-root-a ./translated_a \
+  --md-translated-root-b ./translated_b \
+  --lang zh
+```
+
+</details>
+
+<details>
+<summary><strong>5. Recognize: OCR Post-Processing</strong></summary>
 
 Tools to clean up raw outputs from OCR engines like MinerU.
 
@@ -254,6 +305,10 @@ Tools to clean up raw outputs from OCR engines like MinerU.
 - Unpack Images: extract Base64 images back to files.
 - Organize: flatten nested OCR output directories.
 - Fix: apply OCR fixes and rumdl formatting during organize, or as a standalone step.
+- Fix JSON: apply the same fixes to markdown fields inside paper JSON outputs.
+- Fix Math: validate and repair LaTeX formulas with optional LLM assistance.
+- Fix Mermaid: validate and repair Mermaid diagrams (requires `mmdc` from mermaid-cli).
+- Recommended order: `fix` -> `fix-math` -> `fix-mermaid` -> `fix`.
 
 ```bash
 uv run deepresearch-flow recognize md embed --input ./raw_ocr --output ./clean_md
@@ -275,6 +330,25 @@ uv run deepresearch-flow recognize fix \
 uv run deepresearch-flow recognize fix \
   --input ./ocr_md \
   --in-place
+
+# Fix JSON outputs in place
+uv run deepresearch-flow recognize fix \
+  --json \
+  --input ./paper_outputs \
+  --in-place
+
+# Fix LaTeX formulas in markdown
+uv run deepresearch-flow recognize fix-math \
+  --input ./docs \
+  --model openai/gpt-4o-mini \
+  --in-place
+
+# Fix Mermaid diagrams in JSON outputs
+uv run deepresearch-flow recognize fix-mermaid \
+  --json \
+  --input ./paper_outputs \
+  --model openai/gpt-4o-mini \
+  --in-place
 ```
 
 </details>

{deepresearch_flow-0.3.0 → deepresearch_flow-0.4.1}/README.md

@@ -60,6 +60,7 @@ DeepResearch Flow provides a unified pipeline to **Repair**, **Translate**, **Ex
 - **Smart Extraction**: Turn unstructured Markdown into schema-enforced JSON (summaries, metadata, Q&A) using LLMs (OpenAI, Claude, Gemini, etc.).
 - **Precision Translation**: Translate OCR Markdown to Chinese/Japanese (`.zh.md`, `.ja.md`) while **freezing** formulas, code, tables, and references. No more broken layout.
 - **Local Knowledge DB**: A high-performance local Web UI to browse papers with **Split View** (Source vs. Translated vs. Summary), full-text search, and multi-dimensional filtering.
+- **Coverage Compare**: Compare JSON/PDF/Markdown/Translated datasets to find missing artifacts and export CSV reports.
 - **OCR Post-Processing**: Automatically fix broken references (`[1]` -> `[^1]`), merge split paragraphs, and standardize layouts.
 
 ---
@@ -110,7 +111,36 @@ uv run deepresearch-flow translator translate \
   --fix-level moderate
 ```
 
-#### Step 3: Serve Your Database
+#### Step 3: Repair OCR Outputs (Recommended)
+
+Recommended sequence to stabilize markdown before serving:
+
+```bash
+# 1) Fix OCR markdown (auto-detects JSON if inputs are .json)
+uv run deepresearch-flow recognize fix \
+  --input ./docs \
+  --in-place
+
+# 2) Fix LaTeX formulas
+uv run deepresearch-flow recognize fix-math \
+  --input ./docs \
+  --model openai/gpt-4o-mini \
+  --in-place
+
+# 3) Fix Mermaid diagrams
+uv run deepresearch-flow recognize fix-mermaid \
+  --input ./paper_outputs \
+  --json \
+  --model openai/gpt-4o-mini \
+  --in-place
+
+# 4) Fix again to normalize formatting
+uv run deepresearch-flow recognize fix \
+  --input ./docs \
+  --in-place
+```
+
+#### Step 4: Serve Your Database
 
 Launch a local UI to read and manage your papers.
 
@@ -185,7 +215,27 @@ uv run deepresearch-flow paper db serve \
 </details>
 
 <details>
-<summary><strong>4. Recognize: OCR Post-Processing</strong></summary>
+<summary><strong>4. Paper DB Compare: Coverage Audit</strong></summary>
+
+Compare two datasets (A/B) to find missing PDFs, markdowns, translations, or JSON items, with match metadata.
+
+```bash
+uv run deepresearch-flow paper db compare \
+  --input-a ./a.json \
+  --md-root-b ./md_root \
+  --output-csv ./compare.csv
+
+# Compare translated markdowns by language
+uv run deepresearch-flow paper db compare \
+  --md-translated-root-a ./translated_a \
+  --md-translated-root-b ./translated_b \
+  --lang zh
+```
+
+</details>
+
+<details>
+<summary><strong>5. Recognize: OCR Post-Processing</strong></summary>
 
 Tools to clean up raw outputs from OCR engines like MinerU.
 
@@ -193,6 +243,10 @@ Tools to clean up raw outputs from OCR engines like MinerU.
 - Unpack Images: extract Base64 images back to files.
 - Organize: flatten nested OCR output directories.
 - Fix: apply OCR fixes and rumdl formatting during organize, or as a standalone step.
+- Fix JSON: apply the same fixes to markdown fields inside paper JSON outputs.
+- Fix Math: validate and repair LaTeX formulas with optional LLM assistance.
+- Fix Mermaid: validate and repair Mermaid diagrams (requires `mmdc` from mermaid-cli).
+- Recommended order: `fix` -> `fix-math` -> `fix-mermaid` -> `fix`.
 
 ```bash
 uv run deepresearch-flow recognize md embed --input ./raw_ocr --output ./clean_md
@@ -214,6 +268,25 @@ uv run deepresearch-flow recognize fix \
 uv run deepresearch-flow recognize fix \
   --input ./ocr_md \
   --in-place
+
+# Fix JSON outputs in place
+uv run deepresearch-flow recognize fix \
+  --json \
+  --input ./paper_outputs \
+  --in-place
+
+# Fix LaTeX formulas in markdown
+uv run deepresearch-flow recognize fix-math \
+  --input ./docs \
+  --model openai/gpt-4o-mini \
+  --in-place
+
+# Fix Mermaid diagrams in JSON outputs
+uv run deepresearch-flow recognize fix-mermaid \
+  --json \
+  --input ./paper_outputs \
+  --model openai/gpt-4o-mini \
+  --in-place
 ```
 
 </details>

{deepresearch_flow-0.3.0 → deepresearch_flow-0.4.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "deepresearch-flow"
-version = "0.3.0"
+version = "0.4.1"
 description = "Workflow tools for paper extraction, review, and research automation."
 readme = { file = "README.md", content-type = "text/markdown" }
 license = { file = "LICENSE" }
@@ -20,9 +20,10 @@ dependencies = [
     "markdown-it-py>=3.0.0",
     "mdit-py-plugins>=0.4.0",
     "pypdf>=3.0.0",
+    "pylatexenc>=2.10",
     "pybtex>=0.24.0",
     "rich>=13.7.1",
-    "rumdl>=0.0.214",
+    "rumdl>=0.0.218",
     "starlette>=0.37.2",
     "tqdm>=4.66.4",
     "uvicorn>=0.27.1",
@@ -59,4 +60,5 @@ include-package-data = true
 "deepresearch_flow.paper.templates" = ["*.j2"]
 "deepresearch_flow.paper.prompt_templates" = ["*.j2"]
 "deepresearch_flow.paper.examples" = ["*.toml"]
-"deepresearch_flow.paper.web" = ["pdfjs/**/*"]
+"deepresearch_flow.paper.web" = ["pdfjs/**/*", "templates/**/*", "static/**/*"]
+"deepresearch_flow.recognize" = ["*.js"]

{deepresearch_flow-0.3.0 → deepresearch_flow-0.4.1}/python/deepresearch_flow/paper/db.py

@@ -808,3 +808,187 @@ def register_db_commands(db_group: click.Group) -> None:
             raise click.ClickException(str(exc)) from exc
         rendered = render_papers(papers, out_dir, template, output_language)
         click.echo(f"Rendered {rendered} markdown files")
+
+    @db_group.command("compare")
+    @click.option(
+        "-ia", "--input-a", "input_paths_a", multiple=True, help="Input JSON files for side A (repeatable)"
+    )
+    @click.option(
+        "-ib", "--input-b", "input_paths_b", multiple=True, help="Input JSON files for side B (repeatable)"
+    )
+    @click.option(
+        "--pdf-root-a", "pdf_roots_a", multiple=True, help="PDF root directories for side A (repeatable)"
+    )
+    @click.option(
+        "--pdf-root-b", "pdf_roots_b", multiple=True, help="PDF root directories for side B (repeatable)"
+    )
+    @click.option(
+        "--md-root-a", "md_roots_a", multiple=True, help="Markdown root directories for side A (repeatable)"
+    )
+    @click.option(
+        "--md-root-b", "md_roots_b", multiple=True, help="Markdown root directories for side B (repeatable)"
+    )
+    @click.option(
+        "--md-translated-root-a", "md_translated_roots_a", multiple=True,
+        help="Translated Markdown root directories for side A (repeatable)"
+    )
+    @click.option(
+        "--md-translated-root-b", "md_translated_roots_b", multiple=True,
+        help="Translated Markdown root directories for side B (repeatable)"
+    )
+    @click.option("-b", "--bibtex", "bibtex_path", default=None, help="Optional BibTeX file path")
+    @click.option("--lang", "lang", default=None, help="Language code for translated comparisons (e.g., zh)")
+    @click.option(
+        "--output-csv", "output_csv", default=None, help="Path to export results as CSV"
+    )
+    @click.option(
+        "--sample-limit", "sample_limit", default=5, type=int, show_default=True,
+        help="Number of sample items to show in terminal output"
+    )
+    def compare(
+        input_paths_a: tuple[str, ...],
+        input_paths_b: tuple[str, ...],
+        pdf_roots_a: tuple[str, ...],
+        pdf_roots_b: tuple[str, ...],
+        md_roots_a: tuple[str, ...],
+        md_roots_b: tuple[str, ...],
+        md_translated_roots_a: tuple[str, ...],
+        md_translated_roots_b: tuple[str, ...],
+        bibtex_path: str | None,
+        lang: str | None,
+        output_csv: str | None,
+        sample_limit: int,
+    ) -> None:
+        """Compare two datasets and report matches and differences."""
+        from deepresearch_flow.paper.db_ops import compare_datasets
+        import csv
+        
+        # Validate that at least one input is provided for each side
+        has_input_a = bool(input_paths_a or pdf_roots_a or md_roots_a or md_translated_roots_a)
+        has_input_b = bool(input_paths_b or pdf_roots_b or md_roots_b or md_translated_roots_b)
+        
+        if not has_input_a:
+            raise click.ClickException(
+                "Side A must have at least one input: --input-a, --pdf-root-a, --md-root-a, or --md-translated-root-a"
+            )
+        if not has_input_b:
+            raise click.ClickException(
+                "Side B must have at least one input: --input-b, --pdf-root-b, --md-root-b, or --md-translated-root-b"
+            )
+        if (md_translated_roots_a or md_translated_roots_b) and not lang:
+            raise click.ClickException("--lang is required when comparing translated Markdown datasets")
+        
+        # Run comparison
+        try:
+            results = compare_datasets(
+                json_paths_a=[Path(p) for p in input_paths_a],
+                pdf_roots_a=[Path(p) for p in pdf_roots_a],
+                md_roots_a=[Path(p) for p in md_roots_a],
+                md_translated_roots_a=[Path(p) for p in md_translated_roots_a],
+                json_paths_b=[Path(p) for p in input_paths_b],
+                pdf_roots_b=[Path(p) for p in pdf_roots_b],
+                md_roots_b=[Path(p) for p in md_roots_b],
+                md_translated_roots_b=[Path(p) for p in md_translated_roots_b],
+                bibtex_path=Path(bibtex_path) if bibtex_path else None,
+                lang=lang,
+            )
+        except ValueError as exc:
+            raise click.ClickException(str(exc)) from exc
+        
+        # Calculate statistics
+        total_a = sum(1 for r in results if r.side == "A")
+        total_b = sum(1 for r in results if r.side == "B")
+        matched = sum(1 for r in results if r.side == "MATCH")
+        only_in_a = sum(1 for r in results if r.side == "A" and r.match_status == "only_in_A")
+        only_in_b = sum(1 for r in results if r.side == "B" and r.match_status == "only_in_B")
+        
+        console = Console()
+        
+        # Print summary table
+        summary_table = Table(title="Comparison Summary")
+        summary_table.add_column("Metric", style="cyan")
+        summary_table.add_column("Count", style="green", justify="right")
+        summary_table.add_row("Total in A", str(total_a))
+        summary_table.add_row("Total in B", str(total_b))
+        summary_table.add_row("Matched", str(matched))
+        summary_table.add_row("Only in A", str(only_in_a))
+        summary_table.add_row("Only in B", str(only_in_b))
+        console.print(summary_table)
+        
+        # Print match type breakdown
+        match_types: dict[str, int] = {}
+        for r in results:
+            if r.side == "MATCH" and r.match_type:
+                match_types[r.match_type] = match_types.get(r.match_type, 0) + 1
+        
+        if match_types:
+            type_table = Table(title="Match Types")
+            type_table.add_column("Type", style="cyan")
+            type_table.add_column("Count", style="green", justify="right")
+            for match_type, count in sorted(match_types.items(), key=lambda x: x[1], reverse=True):
+                type_table.add_row(match_type, str(count))
+            console.print(type_table)
+        
+        # Print sample results
+        console.print("\n[bold]Sample Results:[/bold]")
+        
+        # Sample matched items
+        matched_samples = [r for r in results if r.side == "MATCH"][:sample_limit]
+        if matched_samples:
+            console.print("\n[green]Matched Items:[/green]")
+            for r in matched_samples:
+                left = (r.title or "")[:60]
+                right = (r.other_title or "")[:60]
+                console.print(
+                    f"  • {left} ↔ {right} (type: {r.match_type}, score: {r.match_score:.2f})"
+                )
+        
+        # Sample only in A
+        only_a_samples = [
+            r for r in results if r.side == "A" and r.match_status == "only_in_A"
+        ][:sample_limit]
+        if only_a_samples:
+            console.print("\n[yellow]Only in A:[/yellow]")
+            for r in only_a_samples:
+                console.print(f"  • {r.title[:60]}...")
+        
+        # Sample only in B
+        only_b_samples = [
+            r for r in results if r.side == "B" and r.match_status == "only_in_B"
+        ][:sample_limit]
+        if only_b_samples:
+            console.print("\n[yellow]Only in B:[/yellow]")
+            for r in only_b_samples:
+                console.print(f"  • {r.title[:60]}...")
+        
+        # Export to CSV if requested
+        if output_csv:
+            output_path = Path(output_csv)
+            output_path.parent.mkdir(parents=True, exist_ok=True)
+            
+            with open(output_path, "w", newline="", encoding="utf-8") as f:
+                writer = csv.writer(f)
+                writer.writerow([
+                    "Side", "Source Hash", "Title", "Match Status", "Match Type",
+                    "Match Score", "Source Path", "Other Source Hash", "Other Title",
+                    "Other Source Path", "Lang"
+                ])
+                for r in results:
+                    writer.writerow([
+                        r.side,
+                        r.source_hash,
+                        r.title,
+                        r.match_status,
+                        r.match_type or "",
+                        f"{r.match_score:.4f}",
+                        r.source_path or "",
+                        r.other_source_hash or "",
+                        r.other_title or "",
+                        r.other_source_path or "",
+                        r.lang or "",
+                    ])
+            
+            console.print(f"\n[green]Results exported to: {output_path}[/green]")
+        
+        # Print final counts
+        console.print(f"\nTotal results: {len(results)}")