paper-parser-skill 0.1.1__tar.gz

paper_parser_skill-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 KaiHangYang
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

paper_parser_skill-0.1.1/PKG-INFO

@@ -0,0 +1,116 @@
+Metadata-Version: 2.4
+Name: paper-parser-skill
+Version: 0.1.1
+Summary: A CLI tool for searching, downloading, parsing, and summarizing academic papers.
+Author-email: kaihang <kaihang.noir@gmail.com>
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests
+Requires-Dist: click
+Requires-Dist: PyYAML
+Requires-Dist: arxiv
+Requires-Dist: rapidfuzz
+Dynamic: license-file
+
+# Paper Parser 🛠️
+
+**Efficient arXiv Search, Download, and AI-Friendly Markdown Parsing.**
+
+`paper-parser` is a CLI tool designed to streamline the academic research workflow. It handles everything from finding a paper on arXiv to converting it into a clean, structured Markdown format that is optimized for LLMs and AI agents.
+
+## 🚀 Why Use Paper Parser?
+
+Standard PDF-to-text tools often produce one massive block of text, which leads to two major problems when working with AI:
+1.  **Context Overflow**: Large papers can exceed an LLM's context window.
+2.  **Token Waste**: Paying for the entire paper's context when you only need to analyze the "Methodology" or "Conclusion" is expensive and slow.
+
+**The Solution:** `paper-parser` uses the **MinerU V4 API** to extract high-quality Markdown and then **automatically splits the paper into chapters**. This allows AI agents to read the paper **section-by-section**, enabling:
+-   ✅ **Granular Context Management**: Only read what matters.
+-   ✅ **Significant Token Savings**: Drastically reduce your API bills.
+-   ✅ **Higher Accuracy**: Focus the model's attention on specific sections.
+
+---
+
+## ✨ Key Features
+
+-   **🔍 Intelligent Search**: Typos? No problem. Fuzzy-searches arXiv with relevance ranking.
+-   **📥 Smart Download**: Downloads PDFs into organized, ID-based directories.
+-   **🧩 Section Splitting**: Automatically splits papers into `01_Introduction.md`, `02_Methodology.md`, etc.
+-   **📦 Incremental Processing**: Remembers what you've already downloaded and parsed—no redundant API calls.
+-   **🖼️ Image Extraction**: Extracts images and maintains correct relative links within the Markdown chapters.
+-   **📝 Note Templates**: Automatically generates `title.md` and `summary.md` for your research notes.
+
+---
+
+## 🛠️ Installation
+
+### From PyPI (Recommended)
+```bash
+pip install paper-parser-skill
+```
+
+### From Source
+```bash
+# Clone the repository
+git clone https://github.com/KaiHangYang/paper-parser-skill.git
+cd paper-parser-skill
+
+# Install in editable mode
+pip install -e .
+```
+
+## ⚙️ Configuration
+
+The first time you run `pp`, it will create a configuration file at `~/.paper-parser/config.yaml`.
+
+```yaml
+MINERU_API_TOKEN: "your_token_from_mineru.net"
+PAPER_WORKSPACE: "~/paper-parser-workspace"
+MINERU_API_TIMEOUT: 600
+```
+> [!IMPORTANT]
+> You need an API token from [MinerU](https://mineru.net/) to use the parsing features.
+
+---
+
+## 📖 Usage Guide
+
+```bash
+# 1. Search for a paper
+pp search "LLaMA 3"
+
+# 2. Complete workflow: Search -> Download -> Parse -> Meta
+pp all "2303.17564"
+
+# 3. Parse a local PDF file
+pp parse ./my_local_paper.pdf
+
+# 4. Find where a paper is stored
+pp path "LLaMA"
+```
+
+## 📂 Output Structure
+
+```text
+PAPER_WORKSPACE/
+└── 2303.17564/              # ArXiv ID
+    ├── paper.pdf            # Original PDF
+    ├── title.md             # Paper metadata
+    ├── summary.md           # Note-taking template
+    └── markdowns/           # AI-Ready Content
+        ├── 01_Introduction.md
+        ├── 02_Methods.md
+        ├── ...
+        └── images/          # Extracted figures & tables
+```
+
+## 🤝 Acknowledgments
+
+- [arXiv](https://arxiv.org/) for the academic paper API.
+- [RapidFuzz](https://github.com/rapidfuzz/RapidFuzz) for fast fuzzy string matching.
+- [MinerU](https://github.com/opendatalab/MinerU) ([mineru.net](https://mineru.net/)) for high-quality PDF-to-Markdown parsing.
+
+## 📜 License
+
+[MIT](LICENSE)

paper_parser_skill-0.1.1/README.md

@@ -0,0 +1,101 @@
+# Paper Parser 🛠️
+
+**Efficient arXiv Search, Download, and AI-Friendly Markdown Parsing.**
+
+`paper-parser` is a CLI tool designed to streamline the academic research workflow. It handles everything from finding a paper on arXiv to converting it into a clean, structured Markdown format that is optimized for LLMs and AI agents.
+
+## 🚀 Why Use Paper Parser?
+
+Standard PDF-to-text tools often produce one massive block of text, which leads to two major problems when working with AI:
+1.  **Context Overflow**: Large papers can exceed an LLM's context window.
+2.  **Token Waste**: Paying for the entire paper's context when you only need to analyze the "Methodology" or "Conclusion" is expensive and slow.
+
+**The Solution:** `paper-parser` uses the **MinerU V4 API** to extract high-quality Markdown and then **automatically splits the paper into chapters**. This allows AI agents to read the paper **section-by-section**, enabling:
+-   ✅ **Granular Context Management**: Only read what matters.
+-   ✅ **Significant Token Savings**: Drastically reduce your API bills.
+-   ✅ **Higher Accuracy**: Focus the model's attention on specific sections.
+
+---
+
+## ✨ Key Features
+
+-   **🔍 Intelligent Search**: Typos? No problem. Fuzzy-searches arXiv with relevance ranking.
+-   **📥 Smart Download**: Downloads PDFs into organized, ID-based directories.
+-   **🧩 Section Splitting**: Automatically splits papers into `01_Introduction.md`, `02_Methodology.md`, etc.
+-   **📦 Incremental Processing**: Remembers what you've already downloaded and parsed—no redundant API calls.
+-   **🖼️ Image Extraction**: Extracts images and maintains correct relative links within the Markdown chapters.
+-   **📝 Note Templates**: Automatically generates `title.md` and `summary.md` for your research notes.
+
+---
+
+## 🛠️ Installation
+
+### From PyPI (Recommended)
+```bash
+pip install paper-parser-skill
+```
+
+### From Source
+```bash
+# Clone the repository
+git clone https://github.com/KaiHangYang/paper-parser-skill.git
+cd paper-parser-skill
+
+# Install in editable mode
+pip install -e .
+```
+
+## ⚙️ Configuration
+
+The first time you run `pp`, it will create a configuration file at `~/.paper-parser/config.yaml`.
+
+```yaml
+MINERU_API_TOKEN: "your_token_from_mineru.net"
+PAPER_WORKSPACE: "~/paper-parser-workspace"
+MINERU_API_TIMEOUT: 600
+```
+> [!IMPORTANT]
+> You need an API token from [MinerU](https://mineru.net/) to use the parsing features.
+
+---
+
+## 📖 Usage Guide
+
+```bash
+# 1. Search for a paper
+pp search "LLaMA 3"
+
+# 2. Complete workflow: Search -> Download -> Parse -> Meta
+pp all "2303.17564"
+
+# 3. Parse a local PDF file
+pp parse ./my_local_paper.pdf
+
+# 4. Find where a paper is stored
+pp path "LLaMA"
+```
+
+## 📂 Output Structure
+
+```text
+PAPER_WORKSPACE/
+└── 2303.17564/              # ArXiv ID
+    ├── paper.pdf            # Original PDF
+    ├── title.md             # Paper metadata
+    ├── summary.md           # Note-taking template
+    └── markdowns/           # AI-Ready Content
+        ├── 01_Introduction.md
+        ├── 02_Methods.md
+        ├── ...
+        └── images/          # Extracted figures & tables
+```
+
+## 🤝 Acknowledgments
+
+- [arXiv](https://arxiv.org/) for the academic paper API.
+- [RapidFuzz](https://github.com/rapidfuzz/RapidFuzz) for fast fuzzy string matching.
+- [MinerU](https://github.com/opendatalab/MinerU) ([mineru.net](https://mineru.net/)) for high-quality PDF-to-Markdown parsing.
+
+## 📜 License
+
+[MIT](LICENSE)

paper_parser_skill-0.1.1/paper_parser/__init__.py

@@ -0,0 +1,2 @@
+# paper_reader package initialization
+from .config import config

paper_parser_skill-0.1.1/paper_parser/arxiv_client.py

@@ -0,0 +1,65 @@
+import arxiv
+from rapidfuzz import fuzz
+import requests
+from pathlib import Path
+
+def search_arxiv(query, max_results=1):
+    """
+    Search for papers on arXiv with fuzzy matching support.
+    Broadens the search and ranks results by similarity to query.
+    """
+    client = arxiv.Client()
+    
+    # 1. Expand the search results pool for better fuzzy matching coverage
+    try:
+        search = arxiv.Search(
+            query=query,
+            max_results=max_results * 5,
+            sort_by=arxiv.SortCriterion.Relevance
+        )
+        
+        results = []
+        for r in client.results(search):
+            paper_id = r.entry_id.split('/')[-1]
+            title = r.title
+            
+            # 2. Calculate fuzzy similarity score between query and title
+            score = fuzz.partial_ratio(query.lower(), title.lower())
+            
+            results.append({
+                'id': paper_id,
+                'title': title,
+                'pdf_url': r.pdf_url,
+                'score': score
+            })
+        
+        # 3. Sort by fuzzy score descending
+        results.sort(key=lambda x: x['score'], reverse=True)
+        
+        # 4. Limit to the requested number of results
+        return results[:max_results]
+        
+    except Exception as e:
+        print(f"arXiv search failed: {e}")
+        return []
+
+def download_pdf(pdf_url, output_path):
+    """Download PDF from URL to local path."""
+    print(f"📥 Downloading: {pdf_url}")
+    try:
+        response = requests.get(pdf_url, stream=True, timeout=60)
+        response.raise_for_status()
+        
+        # Ensure parent directory exists
+        Path(output_path).parent.mkdir(parents=True, exist_ok=True)
+        
+        with open(output_path, 'wb') as f:
+            for chunk in response.iter_content(chunk_size=8192):
+                if chunk:
+                    f.write(chunk)
+        
+        print(f"✅ PDF saved: {output_path}")
+        return True
+    except Exception as e:
+        print(f"❌ Download failed: {e}")
+        return False

paper_parser_skill-0.1.1/paper_parser/cli.py

@@ -0,0 +1,179 @@
+import click
+import os
+from pathlib import Path
+from . import arxiv_client, mineru_client, utils
+from .config import DEFAULT_CONFIG_PATH
+
+@click.group(help=f"""
+Paper Parser CLI - Search, Download, and Parse academic papers.
+
+Configuration is stored in: {DEFAULT_CONFIG_PATH}
+""")
+def cli():
+    pass
+
+@cli.command()
+@click.argument('query')
+@click.option('--limit', default=1, help='Number of results to show.')
+def search(query, limit):
+    """Search for papers on arXiv."""
+    results = arxiv_client.search_arxiv(query, max_results=limit)
+    if not results:
+        click.echo("No papers found.")
+        return
+
+    for i, res in enumerate(results, 1):
+        click.echo(f"{i}. Id: {res['id']}")
+        click.echo(f"   Title: {res['title']}")
+        click.echo(f"   Link: {res['pdf_url']}")
+
+@cli.command()
+@click.argument('query_or_id')
+@click.option('--force', is_flag=True, help='Force re-download even if PDF exists.')
+def download(query_or_id, force):
+    """Download a paper PDF by arXiv ID or query."""
+    click.echo(f"🔍 Finding paper: {query_or_id}")
+    
+    # 1. Resolve paper
+    is_id = arxiv_client.search_arxiv(f"id:{query_or_id}") if "." in query_or_id else []
+    results = is_id if is_id else arxiv_client.search_arxiv(query_or_id, max_results=1)
+    
+    if not results:
+        click.echo("❌ Paper not found.")
+        return
+    
+    paper = results[0]
+    click.echo(f"📄 Found: {paper['title']}")
+    
+    # 2. Setup directory and metadata
+    paper_dir = utils.get_paper_dir(paper['id'])
+    pdf_path = paper_dir / "paper.pdf"
+    title_path = paper_dir / "title.md"
+    title_path.write_text(f"# {paper['title']}\n", encoding='utf-8')
+    
+    summary_path = paper_dir / "summary.md"
+    if not summary_path.exists():
+        summary_path.write_text(f"# Summary: {paper['title']}\n\n## Key Takeaways\n\n- \n", encoding='utf-8')
+    
+    # 3. Download (with cache check)
+    if not force and pdf_path.exists():
+        click.echo(f"⏭️  Skipping Download: PDF already exists in {paper_dir}")
+        return
+
+    if arxiv_client.download_pdf(paper['pdf_url'], pdf_path):
+        click.echo(f"✅ Paper downloaded to: {paper_dir}")
+
+@cli.command()
+@click.argument('target')
+@click.option('--output-dir', help='Force output directory')
+@click.option('--force', is_flag=True, help='Force re-parsing even if results exist.')
+def parse(target, output_dir, force):
+    """Parse a PDF using MinerU API. TARGET can be a local PDF path or an arXiv ID."""
+    pdf_path = None
+    final_output_dir = None
+
+    # Case 1: Local File
+    if os.path.isfile(target):
+        pdf_path = Path(target)
+        final_output_dir = Path(output_dir) if output_dir else pdf_path.parent / "paper"
+        click.echo(f"📄 Local PDF detected: {pdf_path}")
+    
+    # Case 2: arXiv ID (or something else)
+    else:
+        # Resolve ID to folder
+        paper_dir = utils.get_paper_dir(target)
+        pdf_path = paper_dir / "paper.pdf"
+        final_output_dir = Path(output_dir) if output_dir else paper_dir
+        
+        if not pdf_path.exists():
+            click.echo(f"❌ Error: {target} is not a file, and no paper.pdf found in workspace for ID [{target}].")
+            click.echo(f"   Try 'pp download {target}' first.")
+            return
+        
+        click.echo(f"📚 arXiv paper detected: [{target}]")
+
+    # Incremental check
+    if not force and (final_output_dir / "markdowns").exists():
+        click.echo(f"⏭️  Skipping: Results already exist in {final_output_dir}")
+        return
+
+    click.echo(f"🔄 Parsing {pdf_path}...")
+    try:
+        mineru_client.parse_paper(str(pdf_path), str(final_output_dir))
+        click.echo(f"✅ Parsing complete. Results in {final_output_dir}")
+    except ValueError as e:
+        click.echo(str(e))
+    except Exception as e:
+        click.echo(f"❌ Error during parsing: {e}")
+
+@cli.command()
+@click.argument('query_or_id')
+def path(query_or_id):
+    """Find the local path of a processed paper."""
+    click.echo(f"🔍 Locating paper: {query_or_id}")
+    
+    is_id = arxiv_client.search_arxiv(f"id:{query_or_id}") if "." in query_or_id or "/" in query_or_id else []
+    results = is_id if is_id else arxiv_client.search_arxiv(query_or_id, max_results=1)
+    
+    if not results:
+        click.echo("❌ Paper not found on arXiv.")
+        return
+    
+    paper = results[0]
+    paper_dir = utils.get_paper_dir(paper['id'])
+    
+    if paper_dir.exists() and any(paper_dir.iterdir()):
+        click.echo(f"📍 Local path: {paper_dir.absolute()}")
+    else:
+        click.echo(f"❓ Paper found on arXiv ([{paper['id']}]), but not processed locally yet.")
+        click.echo(f"   Use 'pp all \"{paper['id']}\"' to download and parse it.")
+
+@cli.command()
+@click.argument('query_or_id')
+@click.option('--force', is_flag=True, help='Force re-parsing.')
+def all(query_or_id, force):
+    """Run full workflow: Search -> Download -> Parse."""
+    click.echo(f"🚀 Starting full workflow for: {query_or_id}")
+    
+    is_id = arxiv_client.search_arxiv(f"id:{query_or_id}") if "." in query_or_id else []
+    results = is_id if is_id else arxiv_client.search_arxiv(query_or_id, max_results=1)
+    
+    if not results:
+        click.echo("❌ Paper not found.")
+        return
+    
+    paper = results[0]
+    click.echo(f"📄 Found: {paper['title']}")
+    
+    paper_dir = utils.get_paper_dir(paper['id'])
+    pdf_path = paper_dir / "paper.pdf"
+    title_path = paper_dir / "title.md"
+    title_path.write_text(f"# {paper['title']}\n", encoding='utf-8')
+    
+    summary_path = paper_dir / "summary.md"
+    if not summary_path.exists():
+        summary_path.write_text(f"# Summary: {paper['title']}\n\n## Key Takeaways\n\n- \n", encoding='utf-8')
+    
+    # 3. Download (with cache check)
+    if not force and pdf_path.exists():
+        click.echo(f"⏭️  Skipping Download: PDF already exists in {paper_dir}")
+    else:
+        if not arxiv_client.download_pdf(paper['pdf_url'], pdf_path):
+            return
+    
+    # Reuse parse logic via ctx or just repeat here for simplicity
+    if not force and (paper_dir / "markdowns").exists():
+        click.echo(f"⏭️  Skipping Parsing: Results already exist in {paper_dir}")
+    else:
+        click.echo("🔄 Parsing with MinerU API...")
+        try:
+            mineru_client.parse_paper(str(pdf_path), str(paper_dir))
+            click.echo(f"\n🎉 Workflow complete!")
+            click.echo(f"📂 Folder: {paper_dir}")
+        except ValueError as e:
+            click.echo(str(e))
+        except Exception as e:
+            click.echo(f"❌ Error during parsing: {e}")
+
+if __name__ == '__main__':
+    cli()

paper_parser_skill-0.1.1/paper_parser/config.py

@@ -0,0 +1,52 @@
+import os
+import yaml
+from pathlib import Path
+
+DEFAULT_CONFIG_DIR = Path.home() / ".paper-parser"
+DEFAULT_CONFIG_PATH = DEFAULT_CONFIG_DIR / "config.yaml"
+
+DEFAULT_CONFIG = {
+    "PAPER_WORKSPACE": str(Path.home() / "paper-parser-workspace"),
+    "MINERU_API_TOKEN": "",
+    "MINERU_API_BASE_URL": "https://mineru.net/api/v4",
+    "MINERU_API_TIMEOUT": 600
+}
+
+class Config:
+    def __init__(self, config_path=DEFAULT_CONFIG_PATH):
+        self.config_path = Path(config_path)
+        self.data = {}
+        self.load()
+
+    def load(self):
+        """Load config from YAML file, creating it if it doesn't exist."""
+        if not self.config_path.exists():
+            self.create_default_config()
+        
+        try:
+            with open(self.config_path, 'r', encoding='utf-8') as f:
+                self.data = yaml.safe_load(f) or {}
+        except Exception as e:
+            print(f"Error loading config from {self.config_path}: {e}")
+            self.data = {}
+
+        # Merge with defaults for missing keys
+        for key, value in DEFAULT_CONFIG.items():
+            if key not in self.data:
+                self.data[key] = value
+
+    def create_default_config(self):
+        """Create the config directory and a default config file."""
+        self.config_path.parent.mkdir(parents=True, exist_ok=True)
+        try:
+            with open(self.config_path, 'w', encoding='utf-8') as f:
+                yaml.dump(DEFAULT_CONFIG, f, default_flow_style=False, allow_unicode=True)
+            print(f"Created default config at {self.config_path}")
+        except Exception as e:
+            print(f"Error creating default config: {e}")
+
+    def get(self, key, default=None):
+        return self.data.get(key, default)
+
+# Global config instance
+config = Config()

paper_parser_skill-0.1.1/paper_parser/mineru_client.py

@@ -0,0 +1,202 @@
+import os
+import time
+import requests
+import zipfile
+import io
+import re
+import shutil
+from pathlib import Path
+from .config import config
+
+class MinerUClient:
+    def __init__(self, token=None, base_url=None):
+        self.token = token or config.get("MINERU_API_TOKEN")
+        self.base_url = base_url or config.get("MINERU_API_BASE_URL")
+        
+    def _validate_token(self):
+        if not self.token:
+            raise ValueError(
+                "❌ MinerU API Token is missing!\n"
+                "Please configure it in ~/.paper-parser/config.yaml\n"
+                "Set the 'MINERU_API_TOKEN' field."
+            )
+
+    def _get_headers(self):
+        return {"Authorization": f"Bearer {self.token}"}
+
+    def upload_pdf(self, pdf_path):
+        """Step 1 & 2: Get upload URL and upload PDF."""
+        self._validate_token()
+        
+        url = f"{self.base_url}/file-urls/batch"
+        file_name = os.path.basename(pdf_path)
+        data = {
+            "files": [
+                {"name": file_name, "data_id": str(int(time.time()))}
+            ],
+            "model_version": "vlm"
+        }
+        
+        print(f"[*] Requesting upload URL for {file_name}...")
+        response = requests.post(url, headers=self._get_headers(), json=data)
+        response.raise_for_status()
+        
+        result = response.json()
+        if result.get("code") != 0:
+            raise Exception(f"API Error: {result.get('msg')}")
+            
+        upload_url = result["data"]["file_urls"][0]
+        batch_id = result["data"]["batch_id"]
+        
+        print(f"[*] Uploading {pdf_path}...")
+        with open(pdf_path, "rb") as f:
+            resp = requests.put(upload_url, data=f)
+            resp.raise_for_status()
+        print("[+] Upload successful.")
+        
+        return batch_id
+
+    def poll_status(self, batch_id):
+        """Step 3: Poll for completion."""
+        url = f"{self.base_url}/extract-results/batch/{batch_id}"
+        timeout = config.get("MINERU_API_TIMEOUT", 600)
+        start_time = time.time()
+        
+        print(f"[*] Waiting for conversion to complete (Timeout: {timeout}s)...")
+        while True:
+            elapsed = time.time() - start_time
+            if elapsed > timeout:
+                raise TimeoutError(f"❌ MinerU conversion timed out after {timeout} seconds.")
+
+            response = requests.get(url, headers=self._get_headers())
+            response.raise_for_status()
+            
+            result = response.json()
+            if result.get("code") != 0:
+                raise Exception(f"Status Check Error: {result.get('msg')}")
+                
+            extract_results = result["data"].get("extract_result", [])
+            if not extract_results:
+                print(f"[.] Still processing (queueing)... {int(elapsed)}s elapsed")
+                time.sleep(5)
+                continue
+                
+            file_state = extract_results[0]
+            state = file_state.get("state")
+            
+            if state == "done":
+                print("[+] Conversion finished!")
+                return file_state.get("full_zip_url")
+            elif state == "failed":
+                raise Exception("Conversion failed on server side.")
+            else:
+                print(f"[.] Current state: {state}. Polling in 5s... ({int(elapsed)}s elapsed)")
+                time.sleep(5)
+
+    def process_results(self, zip_url, output_dir):
+        """Step 4: Download and process result ZIP."""
+        print(f"[*] Downloading results...")
+        response = requests.get(zip_url)
+        response.raise_for_status()
+        
+        paper_dir = Path(output_dir)
+        paper_dir.mkdir(parents=True, exist_ok=True)
+        markdowns_dir = paper_dir / "markdowns"
+        markdowns_dir.mkdir(exist_ok=True)
+        images_dir = markdowns_dir / "images"
+        images_dir.mkdir(exist_ok=True)
+        
+        # Temp directory for extraction
+        temp_dir = paper_dir / "_temp_mineru"
+        if temp_dir.exists():
+            shutil.rmtree(temp_dir)
+        temp_dir.mkdir()
+        temp_images_dir = temp_dir / "images"
+        temp_images_dir.mkdir()
+        
+        md_content = ""
+        with zipfile.ZipFile(io.BytesIO(response.content)) as z:
+            for name in z.namelist():
+                if name.endswith("full.md"):
+                    md_content = z.read(name).decode("utf-8")
+                elif "images/" in name and not name.endswith("/"):
+                    filename = os.path.basename(name)
+                    if filename:
+                        with z.open(name) as source, open(temp_images_dir / filename, "wb") as target:
+                            shutil.copyfileobj(source, target)
+
+        if not md_content:
+            shutil.rmtree(temp_dir)
+            raise Exception("Error: Could not find 'full.md' in the result ZIP.")
+        
+        # Process images and references
+        processed_md = self._process_images(md_content, temp_images_dir, images_dir)
+        
+        # Split into chapters
+        chapter_count = self._split_chapters(processed_md, markdowns_dir)
+        
+        # Cleanup
+        shutil.rmtree(temp_dir)
+        print(f"[+] Done! {chapter_count} chapters saved to 'markdowns/'. Images saved to 'images/'.")
+        return chapter_count
+
+    def _process_images(self, md_content, temp_images_dir, final_images_dir):
+        md_img_pattern = re.compile(r'!\[([^\]]*)\]\(images/([^)]+)\)')
+        html_img_pattern = re.compile(r'<img[^>]+src="images/([^"]+)"[^>]*>')
+        
+        unique_refs = []
+        for match in md_img_pattern.finditer(md_content):
+            if match.group(2) not in unique_refs:
+                unique_refs.append(match.group(2))
+        for match in html_img_pattern.finditer(md_content):
+            if match.group(1) not in unique_refs:
+                unique_refs.append(match.group(1))
+                
+        mapping = {}
+        for i, old_name in enumerate(unique_refs, 1):
+            new_name = f"{i}.jpg"
+            old_path = temp_images_dir / old_name
+            new_path = final_images_dir / new_name
+            
+            if old_path.exists():
+                shutil.move(str(old_path), str(new_path))
+                mapping[old_name] = new_name
+
+        def md_replace(match):
+            alt, name = match.groups()
+            return f"![{alt}](images/{mapping.get(name, name)})"
+            
+        def html_replace(match):
+            name = match.group(1)
+            return f'<img src="images/{mapping.get(name, name)}" />'
+
+        md_content = md_img_pattern.sub(md_replace, md_content)
+        md_content = html_img_pattern.sub(html_replace, md_content)
+        return md_content
+
+    def _split_chapters(self, md_content, output_folder):
+        header_pattern = re.compile(r'^(#{1,6})\s+(.+)$', re.MULTILINE)
+        headers = [(m.start(), len(m.group(1)), m.group(2).strip()) for m in header_pattern.finditer(md_content)]
+
+        if not headers:
+            (output_folder / "00_Complete.md").write_text(md_content.strip() + '\n', encoding='utf-8')
+            return 1
+
+        for i, (start, level, title) in enumerate(headers):
+            end = headers[i + 1][0] if i + 1 < len(headers) else len(md_content)
+            content = md_content[start:end].strip()
+            safe_title = re.sub(r'[<>:"/\\|?*]', '_', title)
+            safe_title = re.sub(r'\s+', '_', safe_title)[:80]
+            filename = f"{i+1:02d}_{safe_title}.md"
+            (output_folder / filename).write_text(content + '\n', encoding='utf-8')
+        
+        return len(headers)
+
+def parse_paper(pdf_path, output_dir):
+    """Convenience function to run the full MinerU workflow."""
+    client = MinerUClient()
+    batch_id = client.upload_pdf(pdf_path)
+    zip_url = client.poll_status(batch_id)
+    if zip_url:
+        return client.process_results(zip_url, output_dir)
+    return 0

paper_parser_skill-0.1.1/paper_parser/utils.py

@@ -0,0 +1,25 @@
+import os
+import re
+from pathlib import Path
+from .config import config
+
+def sanitize_id(paper_id):
+    """Sanitize arXiv ID to be used as a directory name."""
+    # Replace slashes in older IDs (e.g., hep-th/9901001) with underscores
+    return re.sub(r'[\\/:*?"<>|]', '_', paper_id)
+
+def get_work_dir():
+    """Get the base workspace directory from config."""
+    workspace = config.get("PAPER_WORKSPACE")
+    # Resolve ~ if present
+    workspace_path = Path(os.path.expanduser(workspace))
+    workspace_path.mkdir(parents=True, exist_ok=True)
+    return workspace_path
+
+def get_paper_dir(paper_id):
+    """Get the directory for a specific paper using its ID."""
+    base_dir = get_work_dir()
+    safe_id = sanitize_id(paper_id)
+    paper_dir = base_dir / safe_id
+    paper_dir.mkdir(parents=True, exist_ok=True)
+    return paper_dir

paper_parser_skill-0.1.1/paper_parser_skill.egg-info/PKG-INFO

@@ -0,0 +1,116 @@
+Metadata-Version: 2.4
+Name: paper-parser-skill
+Version: 0.1.1
+Summary: A CLI tool for searching, downloading, parsing, and summarizing academic papers.
+Author-email: kaihang <kaihang.noir@gmail.com>
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests
+Requires-Dist: click
+Requires-Dist: PyYAML
+Requires-Dist: arxiv
+Requires-Dist: rapidfuzz
+Dynamic: license-file
+
+# Paper Parser 🛠️
+
+**Efficient arXiv Search, Download, and AI-Friendly Markdown Parsing.**
+
+`paper-parser` is a CLI tool designed to streamline the academic research workflow. It handles everything from finding a paper on arXiv to converting it into a clean, structured Markdown format that is optimized for LLMs and AI agents.
+
+## 🚀 Why Use Paper Parser?
+
+Standard PDF-to-text tools often produce one massive block of text, which leads to two major problems when working with AI:
+1.  **Context Overflow**: Large papers can exceed an LLM's context window.
+2.  **Token Waste**: Paying for the entire paper's context when you only need to analyze the "Methodology" or "Conclusion" is expensive and slow.
+
+**The Solution:** `paper-parser` uses the **MinerU V4 API** to extract high-quality Markdown and then **automatically splits the paper into chapters**. This allows AI agents to read the paper **section-by-section**, enabling:
+-   ✅ **Granular Context Management**: Only read what matters.
+-   ✅ **Significant Token Savings**: Drastically reduce your API bills.
+-   ✅ **Higher Accuracy**: Focus the model's attention on specific sections.
+
+---
+
+## ✨ Key Features
+
+-   **🔍 Intelligent Search**: Typos? No problem. Fuzzy-searches arXiv with relevance ranking.
+-   **📥 Smart Download**: Downloads PDFs into organized, ID-based directories.
+-   **🧩 Section Splitting**: Automatically splits papers into `01_Introduction.md`, `02_Methodology.md`, etc.
+-   **📦 Incremental Processing**: Remembers what you've already downloaded and parsed—no redundant API calls.
+-   **🖼️ Image Extraction**: Extracts images and maintains correct relative links within the Markdown chapters.
+-   **📝 Note Templates**: Automatically generates `title.md` and `summary.md` for your research notes.
+
+---
+
+## 🛠️ Installation
+
+### From PyPI (Recommended)
+```bash
+pip install paper-parser-skill
+```
+
+### From Source
+```bash
+# Clone the repository
+git clone https://github.com/KaiHangYang/paper-parser-skill.git
+cd paper-parser-skill
+
+# Install in editable mode
+pip install -e .
+```
+
+## ⚙️ Configuration
+
+The first time you run `pp`, it will create a configuration file at `~/.paper-parser/config.yaml`.
+
+```yaml
+MINERU_API_TOKEN: "your_token_from_mineru.net"
+PAPER_WORKSPACE: "~/paper-parser-workspace"
+MINERU_API_TIMEOUT: 600
+```
+> [!IMPORTANT]
+> You need an API token from [MinerU](https://mineru.net/) to use the parsing features.
+
+---
+
+## 📖 Usage Guide
+
+```bash
+# 1. Search for a paper
+pp search "LLaMA 3"
+
+# 2. Complete workflow: Search -> Download -> Parse -> Meta
+pp all "2303.17564"
+
+# 3. Parse a local PDF file
+pp parse ./my_local_paper.pdf
+
+# 4. Find where a paper is stored
+pp path "LLaMA"
+```
+
+## 📂 Output Structure
+
+```text
+PAPER_WORKSPACE/
+└── 2303.17564/              # ArXiv ID
+    ├── paper.pdf            # Original PDF
+    ├── title.md             # Paper metadata
+    ├── summary.md           # Note-taking template
+    └── markdowns/           # AI-Ready Content
+        ├── 01_Introduction.md
+        ├── 02_Methods.md
+        ├── ...
+        └── images/          # Extracted figures & tables
+```
+
+## 🤝 Acknowledgments
+
+- [arXiv](https://arxiv.org/) for the academic paper API.
+- [RapidFuzz](https://github.com/rapidfuzz/RapidFuzz) for fast fuzzy string matching.
+- [MinerU](https://github.com/opendatalab/MinerU) ([mineru.net](https://mineru.net/)) for high-quality PDF-to-Markdown parsing.
+
+## 📜 License
+
+[MIT](LICENSE)

paper_parser_skill-0.1.1/paper_parser_skill.egg-info/SOURCES.txt

@@ -0,0 +1,15 @@
+LICENSE
+README.md
+pyproject.toml
+paper_parser/__init__.py
+paper_parser/arxiv_client.py
+paper_parser/cli.py
+paper_parser/config.py
+paper_parser/mineru_client.py
+paper_parser/utils.py
+paper_parser_skill.egg-info/PKG-INFO
+paper_parser_skill.egg-info/SOURCES.txt
+paper_parser_skill.egg-info/dependency_links.txt
+paper_parser_skill.egg-info/entry_points.txt
+paper_parser_skill.egg-info/requires.txt
+paper_parser_skill.egg-info/top_level.txt

paper_parser_skill-0.1.1/paper_parser_skill.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

paper_parser_skill-0.1.1/paper_parser_skill.egg-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+paper-parser = paper_parser.cli:cli
+pp = paper_parser.cli:cli

paper_parser_skill-0.1.1/paper_parser_skill.egg-info/requires.txt

@@ -0,0 +1,5 @@
+requests
+click
+PyYAML
+arxiv
+rapidfuzz

paper_parser_skill-0.1.1/paper_parser_skill.egg-info/top_level.txt

@@ -0,0 +1 @@
+paper_parser

paper_parser_skill-0.1.1/pyproject.toml

@@ -0,0 +1,28 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "paper-parser-skill"
+version = "0.1.1"
+description = "A CLI tool for searching, downloading, parsing, and summarizing academic papers."
+readme = "README.md"
+authors = [
+    { name = "kaihang", email = "kaihang.noir@gmail.com" }
+]
+dependencies = [
+    "requests",
+    "click",
+    "PyYAML",
+    "arxiv",
+    "rapidfuzz",
+]
+requires-python = ">=3.8"
+
+[project.scripts]
+paper-parser = "paper_parser.cli:cli"
+pp = "paper_parser.cli:cli"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["paper_parser*"]

paper_parser_skill-0.1.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+