paper-harvest 0.2.0__tar.gz

paper_harvest-0.2.0/PKG-INFO

@@ -0,0 +1,144 @@
+Metadata-Version: 2.4
+Name: paper-harvest
+Version: 0.2.0
+Summary: Harvest arXiv papers into organized article directories with PDF, TeX, and Markdown.
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+
+# paper-harvest
+
+Collect arXiv papers into a unified article directory: PDF, TeX source, optional Markdown, and a structured manifest.
+
+Python 3.9+, stdlib only at runtime. Two ways to use it:
+
+| Audience | How |
+|----------|-----|
+| Terminal / automation | `uvx paper-harvest` (PyPI) |
+| Cursor / Agent | Install the [skill zip](#cursor-skill) |
+
+## Quick start
+
+```bash
+uvx paper-harvest 'https://arxiv.org/abs/2403.18074' --json
+```
+
+From a local checkout:
+
+```bash
+uvx --from . paper-harvest 'https://arxiv.org/abs/2403.18074' --json
+```
+
+Install permanently:
+
+```bash
+uv tool install paper-harvest
+paper-harvest 'https://arxiv.org/abs/2403.18074' --json
+```
+
+**Input:** arXiv URLs only (`arxiv.org`, `abs` / `pdf` / `src`).
+
+## Output
+
+```text
+<article_id>/
+  pdf/<article_id>.pdf
+  tex/
+    arXiv-<id>.tar.gz
+    source/
+  md/
+    <article_id>.md
+    assets/
+    mineru_result.json
+  harvest_result.json
+```
+
+Each step reports `ok`, `skipped`, or `failed` in `harvest_result.json`. Failed steps do not remove earlier artifacts.
+
+## CLI flags
+
+| Flag | Effect |
+|------|--------|
+| `--target-dir <dir>` | Output root (default: `.`) |
+| `--skip-tex` | Skip TeX source download |
+| `--skip-md` | Skip Markdown extraction |
+| `--force` | Overwrite existing `md/` |
+| `--json` | Print structured JSON to stdout |
+| `--timeout <sec>` | Download timeout (default: 120) |
+
+## Markdown (MinerU)
+
+Markdown is optional and uses [MinerU](https://mineru.net/).
+
+1. **With token** → precision API (better quality, larger PDFs)
+2. **Without token** → lightweight API (PDFs ≤ 10 MB)
+3. **Otherwise** → `md` is `skipped` or `failed`; PDF and TeX are kept
+
+Configure a token via any one of:
+
+```bash
+export MINERU_API_TOKEN='your-token'
+echo 'your-token' > ~/.mineru_token && chmod 600 ~/.mineru_token
+```
+
+Also checked: `./.mineru_token`, `~/.config/mineru/token`. Never commit token files.
+
+Re-run Markdown only after adding a token:
+
+```bash
+uvx paper-harvest '<same-arxiv-url>' --skip-tex --force --json
+```
+
+## Cursor Skill
+
+The skill is Agent instructions only; execution still goes through the CLI.
+
+**Install** — download `paper-harvest-<version>-skill.zip` from releases:
+
+```bash
+unzip paper-harvest-0.2.0-skill.zip -d ~/.cursor/skills/
+```
+
+**Build locally:**
+
+```bash
+uv build
+python tools/build_skill_zip.py
+# → dist/paper-harvest-0.2.0-skill.zip
+```
+
+Skill sources live in `skill/` (`SKILL.md`, `references/`, `scripts/harvest.sh`).
+
+## Project layout
+
+```text
+paper-harvest/
+  pyproject.toml
+  src/paper_harvest/     # CLI + core
+  skill/                   # Agent skill (shipped as zip)
+  docs/spec.md             # product specification
+  tools/build_skill_zip.py
+  tests/
+```
+
+## Development
+
+```bash
+uv sync --extra dev
+uv run pytest
+uv build
+uvx --from . paper-harvest --help
+```
+
+## Publish
+
+1. Bump `version` in `pyproject.toml` and `src/paper_harvest/__init__.py`
+2. `uv run pytest && uv build && python tools/build_skill_zip.py`
+3. Tag `v0.x.y`, push, attach `dist/*-skill.zip` to GitHub Release
+4. `uv publish` for PyPI (`uvx paper-harvest`)
+
+## Docs
+
+- `docs/spec.md` — product / engineering specification
+- `skill/references/` — Agent-facing reference (manifest schema, MinerU setup)

paper_harvest-0.2.0/README.md

@@ -0,0 +1,135 @@
+# paper-harvest
+
+Collect arXiv papers into a unified article directory: PDF, TeX source, optional Markdown, and a structured manifest.
+
+Python 3.9+, stdlib only at runtime. Two ways to use it:
+
+| Audience | How |
+|----------|-----|
+| Terminal / automation | `uvx paper-harvest` (PyPI) |
+| Cursor / Agent | Install the [skill zip](#cursor-skill) |
+
+## Quick start
+
+```bash
+uvx paper-harvest 'https://arxiv.org/abs/2403.18074' --json
+```
+
+From a local checkout:
+
+```bash
+uvx --from . paper-harvest 'https://arxiv.org/abs/2403.18074' --json
+```
+
+Install permanently:
+
+```bash
+uv tool install paper-harvest
+paper-harvest 'https://arxiv.org/abs/2403.18074' --json
+```
+
+**Input:** arXiv URLs only (`arxiv.org`, `abs` / `pdf` / `src`).
+
+## Output
+
+```text
+<article_id>/
+  pdf/<article_id>.pdf
+  tex/
+    arXiv-<id>.tar.gz
+    source/
+  md/
+    <article_id>.md
+    assets/
+    mineru_result.json
+  harvest_result.json
+```
+
+Each step reports `ok`, `skipped`, or `failed` in `harvest_result.json`. Failed steps do not remove earlier artifacts.
+
+## CLI flags
+
+| Flag | Effect |
+|------|--------|
+| `--target-dir <dir>` | Output root (default: `.`) |
+| `--skip-tex` | Skip TeX source download |
+| `--skip-md` | Skip Markdown extraction |
+| `--force` | Overwrite existing `md/` |
+| `--json` | Print structured JSON to stdout |
+| `--timeout <sec>` | Download timeout (default: 120) |
+
+## Markdown (MinerU)
+
+Markdown is optional and uses [MinerU](https://mineru.net/).
+
+1. **With token** → precision API (better quality, larger PDFs)
+2. **Without token** → lightweight API (PDFs ≤ 10 MB)
+3. **Otherwise** → `md` is `skipped` or `failed`; PDF and TeX are kept
+
+Configure a token via any one of:
+
+```bash
+export MINERU_API_TOKEN='your-token'
+echo 'your-token' > ~/.mineru_token && chmod 600 ~/.mineru_token
+```
+
+Also checked: `./.mineru_token`, `~/.config/mineru/token`. Never commit token files.
+
+Re-run Markdown only after adding a token:
+
+```bash
+uvx paper-harvest '<same-arxiv-url>' --skip-tex --force --json
+```
+
+## Cursor Skill
+
+The skill is Agent instructions only; execution still goes through the CLI.
+
+**Install** — download `paper-harvest-<version>-skill.zip` from releases:
+
+```bash
+unzip paper-harvest-0.2.0-skill.zip -d ~/.cursor/skills/
+```
+
+**Build locally:**
+
+```bash
+uv build
+python tools/build_skill_zip.py
+# → dist/paper-harvest-0.2.0-skill.zip
+```
+
+Skill sources live in `skill/` (`SKILL.md`, `references/`, `scripts/harvest.sh`).
+
+## Project layout
+
+```text
+paper-harvest/
+  pyproject.toml
+  src/paper_harvest/     # CLI + core
+  skill/                   # Agent skill (shipped as zip)
+  docs/spec.md             # product specification
+  tools/build_skill_zip.py
+  tests/
+```
+
+## Development
+
+```bash
+uv sync --extra dev
+uv run pytest
+uv build
+uvx --from . paper-harvest --help
+```
+
+## Publish
+
+1. Bump `version` in `pyproject.toml` and `src/paper_harvest/__init__.py`
+2. `uv run pytest && uv build && python tools/build_skill_zip.py`
+3. Tag `v0.x.y`, push, attach `dist/*-skill.zip` to GitHub Release
+4. `uv publish` for PyPI (`uvx paper-harvest`)
+
+## Docs
+
+- `docs/spec.md` — product / engineering specification
+- `skill/references/` — Agent-facing reference (manifest schema, MinerU setup)

paper_harvest-0.2.0/pyproject.toml

@@ -0,0 +1,23 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "paper-harvest"
+version = "0.2.0"
+description = "Harvest arXiv papers into organized article directories with PDF, TeX, and Markdown."
+readme = "README.md"
+requires-python = ">=3.9"
+dependencies = []
+
+[project.scripts]
+paper-harvest = "paper_harvest.cli:main"
+
+[project.optional-dependencies]
+dev = ["pytest>=7"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

paper_harvest-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

paper_harvest-0.2.0/src/paper_harvest/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.2.0"

paper_harvest-0.2.0/src/paper_harvest/cli.py

@@ -0,0 +1,231 @@
+"""Harvest arXiv papers into organized article directories."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+from paper_harvest.core.article_layout import build_article_layout
+from paper_harvest.core.models import HarvestResult, StepResult
+from paper_harvest.core.url.parse import parse_source
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(
+        prog="paper-harvest",
+        description="Harvest arXiv papers into organized article directories.",
+    )
+    parser.add_argument(
+        "source",
+        help="arXiv URL",
+    )
+    parser.add_argument(
+        "--target-dir",
+        default=".",
+        help="Output root directory (default: current directory)",
+    )
+    parser.add_argument(
+        "--skip-tex",
+        action="store_true",
+        help="Skip TeX source download",
+    )
+    parser.add_argument(
+        "--skip-md",
+        action="store_true",
+        help="Skip Markdown extraction",
+    )
+    parser.add_argument(
+        "--force",
+        action="store_true",
+        help="Overwrite existing md/ output",
+    )
+    parser.add_argument(
+        "--json",
+        action="store_true",
+        help="Output structured JSON result to stdout",
+    )
+    parser.add_argument(
+        "--timeout",
+        type=int,
+        default=120,
+        help="Download timeout in seconds (default: 120)",
+    )
+
+    args = parser.parse_args(argv)
+    return run(args)
+
+
+def run(args: argparse.Namespace) -> int:
+    try:
+        source = parse_source(args.source)
+    except ValueError as exc:
+        sys.stderr.write(f"paper-harvest: {exc}\n")
+        return 1
+
+    layout = build_article_layout(args.target_dir, source.article_id_candidate)
+
+    pdf_result = step_download_pdf(source, layout, timeout=args.timeout)
+
+    tex_result: StepResult
+    if args.skip_tex:
+        tex_result = StepResult(status="skipped", message="--skip-tex")
+    else:
+        tex_result = step_fetch_tex(source, layout, timeout=args.timeout)
+
+    md_result: StepResult
+    if args.skip_md:
+        md_result = StepResult(status="skipped", message="--skip-md")
+    else:
+        md_result = step_extract_md(
+            source,
+            layout,
+            pdf_path=Path(pdf_result.path) if pdf_result.path else None,
+            force=args.force,
+        )
+
+    result = HarvestResult(
+        tool_name="paper-harvest",
+        article_id=layout.article_id,
+        article_dir=str(layout.article_dir),
+        source=source,
+        pdf=pdf_result,
+        tex=tex_result,
+        md=md_result,
+        manifest_path=str(layout.article_dir / "harvest_result.json"),
+    )
+
+    manifest = result.to_dict()
+    layout.article_dir.mkdir(parents=True, exist_ok=True)
+    (layout.article_dir / "harvest_result.json").write_text(
+        json.dumps(manifest, ensure_ascii=False, indent=2) + "\n",
+        encoding="utf-8",
+    )
+
+    if args.json:
+        print(json.dumps(manifest, ensure_ascii=False, indent=2))
+    else:
+        print_summary(result)
+
+    failed = any(s.status == "failed" for s in [result.pdf, result.tex, result.md])
+    return 1 if failed else 0
+
+
+def step_download_pdf(
+    source,
+    layout,
+    *,
+    timeout: int,
+) -> StepResult:
+    try:
+        from paper_harvest.core.pdf.download import download_pdf
+
+        dl_result = download_pdf(source, layout, timeout=timeout)
+        return StepResult(
+            status="ok",
+            message="pdf downloaded",
+            path=str(dl_result.pdf_path),
+        )
+    except Exception as exc:
+        return StepResult(
+            status="failed",
+            message=str(exc),
+            error_code=exc_to_code(exc),
+        )
+
+
+def step_fetch_tex(
+    source,
+    layout,
+    *,
+    timeout: int,
+) -> StepResult:
+    try:
+        from paper_harvest.core.tex.arxiv import fetch_arxiv_source
+
+        tex_result = fetch_arxiv_source(source, layout, timeout=timeout)
+        return StepResult(
+            status="ok",
+            message="tex source fetched",
+            path=str(tex_result.extracted_dir),
+            artifacts=[str(tex_result.archive_path)],
+        )
+    except Exception as exc:
+        return StepResult(
+            status="failed",
+            message=str(exc),
+            error_code=exc_to_code(exc),
+        )
+
+
+def step_extract_md(
+    source,
+    layout,
+    *,
+    pdf_path: Path | None,
+    force: bool,
+) -> StepResult:
+    if pdf_path is None or not pdf_path.exists():
+        return StepResult(
+            status="failed",
+            message="no PDF available for Markdown extraction",
+        )
+
+    try:
+        from paper_harvest.core.md.mineru import MineruError, extract_markdown
+
+        md_result = extract_markdown(
+            pdf_path,
+            layout,
+            source,
+            force=force,
+        )
+        return StepResult(
+            status="ok",
+            message="markdown extracted",
+            path=str(md_result.markdown_path),
+            artifacts=[str(md_result.markdown_path)],
+        )
+    except MineruError as exc:
+        code = exc.error_code or exc_to_code(exc)
+        if code in ("-30001",):
+            return StepResult(
+                status="skipped",
+                message=f"lightweight API not applicable: {exc}",
+                error_code=code,
+            )
+        return StepResult(
+            status="failed",
+            message=str(exc),
+            error_code=code,
+        )
+    except Exception as exc:
+        return StepResult(
+            status="failed",
+            message=str(exc),
+            error_code=exc_to_code(exc),
+        )
+
+
+def exc_to_code(exc: Exception) -> str:
+    return type(exc).__name__
+
+
+def print_summary(result: HarvestResult) -> None:
+    print(f"article  : {result.article_id}")
+    print(f"directory: {result.article_dir}")
+    print(f"pdf      : [{result.pdf.status}] {result.pdf.message}")
+    if result.pdf.path:
+        print(f"           {result.pdf.path}")
+    print(f"tex      : [{result.tex.status}] {result.tex.message}")
+    if result.tex.path:
+        print(f"           {result.tex.path}")
+    print(f"md       : [{result.md.status}] {result.md.message}")
+    if result.md.path:
+        print(f"           {result.md.path}")
+    print(f"manifest : {result.manifest_path}")
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

paper_harvest-0.2.0/src/paper_harvest/core/article_layout.py

@@ -0,0 +1,67 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+import re
+
+
+@dataclass(frozen=True)
+class ArticleLayout:
+    article_id: str
+    target_dir: Path
+    article_dir: Path
+    pdf_dir: Path
+    md_dir: Path
+    tex_dir: Path
+
+
+def normalize_article_id(value: str) -> str:
+    normalized = re.sub(r"[^A-Za-z0-9]+", "_", value).strip("_").lower()
+    if not normalized:
+        raise ValueError("article id is empty")
+    return normalized
+
+
+def build_article_layout(output_root: str | Path, article_id: str) -> ArticleLayout:
+    normalized_id = normalize_article_id(article_id)
+    target_dir = Path(output_root).expanduser().resolve()
+    article_dir = target_dir / normalized_id
+    return ArticleLayout(
+        article_id=normalized_id,
+        target_dir=target_dir,
+        article_dir=article_dir,
+        pdf_dir=article_dir / "pdf",
+        md_dir=article_dir / "md",
+        tex_dir=article_dir / "tex",
+    )
+
+
+def ensure_variant_dir(layout: ArticleLayout, variant: str) -> Path:
+    if variant == "pdf":
+        target_dir = layout.pdf_dir
+    elif variant == "md":
+        target_dir = layout.md_dir
+    elif variant == "tex":
+        target_dir = layout.tex_dir
+    else:
+        raise ValueError(f"Unsupported article variant: {variant}")
+
+    target_dir.mkdir(parents=True, exist_ok=True)
+    return target_dir
+
+
+def ensure_article_layout(layout: ArticleLayout) -> ArticleLayout:
+    layout.article_dir.mkdir(parents=True, exist_ok=True)
+    layout.pdf_dir.mkdir(parents=True, exist_ok=True)
+    layout.md_dir.mkdir(parents=True, exist_ok=True)
+    layout.tex_dir.mkdir(parents=True, exist_ok=True)
+    return layout
+
+
+def to_public_path(path: str | Path, target_dir: str | Path) -> str:
+    resolved_path = Path(path).expanduser().resolve()
+    resolved_target_dir = Path(target_dir).expanduser().resolve()
+    try:
+        return str(resolved_path.relative_to(resolved_target_dir))
+    except ValueError:
+        return str(resolved_path)