acatome-extract 0.2.0__tar.gz

acatome_extract-0.2.0/.github/workflows/publish.yml

@@ -0,0 +1,33 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  id-token: write  # trusted publishing (OIDC)
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v4
+        with:
+          version: "latest"
+      - run: uv venv
+      - run: uv pip install -e ".[dev]"
+      - run: uv run --no-sync pytest
+
+  publish:
+    needs: test
+    runs-on: ubuntu-latest
+    environment: pypi
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v4
+        with:
+          version: "latest"
+      - run: uv build
+      - uses: pypa/gh-action-pypi-publish@release/v1

acatome_extract-0.2.0/.gitignore

@@ -0,0 +1,37 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+*.egg
+dist/
+build/
+
+# Virtual environments
+.venv/
+venv/
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db
+
+# UV
+uv.lock
+
+# Data
+*.acatome/
+inbox/
+completed/
+errors/

acatome_extract-0.2.0/CHANGELOG.md

@@ -0,0 +1,11 @@
+# Changelog
+
+All notable changes to **acatome-extract** will be documented in this file.
+
+Format follows [Keep a Changelog](https://keepachangelog.com/).
+
+## [0.1.0] — 2026-03-11
+
+### Added
+
+- Initial release.

acatome_extract-0.2.0/LICENSE

@@ -0,0 +1,14 @@
+Copyright (c) 2026 Reto Stamm and Acatome Contributors
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.

acatome_extract-0.2.0/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: acatome-extract
+Version: 0.2.0
+Summary: PDF extraction pipeline for scientific papers
+Project-URL: Homepage, https://github.com/acatome/acatome-extract
+Project-URL: Repository, https://github.com/acatome/acatome-extract
+Project-URL: Issues, https://github.com/acatome/acatome-extract/issues
+Author-email: Reto Stamm <reto@retostamm.com>
+License-Expression: GPL-3.0-or-later
+License-File: LICENSE
+Keywords: embeddings,extraction,nlp,pdf,scientific-papers,summarization
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.11
+Requires-Dist: acatome-meta>=0.1.0
+Requires-Dist: litellm>=1.40
+Requires-Dist: marker-pdf>=1.0
+Requires-Dist: precis-summary>=0.1.0
+Requires-Dist: typer>=0.12
+Requires-Dist: watchdog>=4.0
+Provides-Extra: dev
+Requires-Dist: black>=24.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Provides-Extra: embeddings
+Requires-Dist: sentence-transformers>=3.0; extra == 'embeddings'
+Provides-Extra: gpu
+Requires-Dist: sentence-transformers>=3.0; extra == 'gpu'
+Requires-Dist: torch>=2.0; extra == 'gpu'
+Provides-Extra: grobid
+Requires-Dist: grobid-client-python>=0.0.7; extra == 'grobid'
+Provides-Extra: store
+Requires-Dist: acatome-store>=0.1.0; extra == 'store'
+Description-Content-Type: text/markdown
+
+# acatome-extract
+
+PDF extraction and enrichment pipeline for scientific papers. Converts PDFs into structured, searchable bundles with block-level summaries and embeddings.
+
+## Features
+
+- **Marker PDF extraction** — structured block extraction with headings, tables, figures
+- **Fitz fallback** — recursive character chunking when Marker is unavailable
+- **LLM enrichment** — block and paper summaries via Ollama or litellm
+- **Embeddings** — sentence-transformer embeddings for semantic search
+- **File watcher** — `acatome-extract watch` monitors an inbox folder
+- **Bundle format** — `.acatome` companion files for sharing pre-built extractions
+- **CLI** — `acatome-extract` command for extract, enrich, and watch workflows
+
+## Installation
+
+```bash
+uv pip install -e .
+```
+
+With GPU acceleration:
+
+```bash
+uv pip install -e ".[gpu]"
+```
+
+## Usage
+
+```python
+from acatome_extract.pipeline import extract
+
+bundle = extract("/path/to/paper.pdf")
+```
+
+## CLI
+
+```bash
+# Extract (RAKE summaries included automatically, no LLM needed)
+acatome-extract extract paper.pdf
+acatome-extract extract --type datasheet TI_LM317.pdf   # non-article types
+
+# Enrich — embeddings only by default; add --summarize for LLM summaries
+acatome-extract enrich /path/to/bundle
+acatome-extract enrich --summarize /path/to/bundle       # enable LLM summaries
+acatome-extract enrich --summarize --skip-existing dir/   # incremental LLM pass
+
+# Watch — extract + embed + ingest; LLM summaries off by default
+acatome-extract watch ~/papers/inbox
+acatome-extract watch ~/papers/inbox --summarize          # enable LLM summaries
+
+# Migrate old bundles to new summaries dict format + add RAKE
+acatome-extract migrate ~/.acatome/papers
+acatome-extract migrate ~/.acatome/papers --dry-run       # preview changes
+
+# Supplements
+acatome-extract attach parent-slug supplement.pdf --name s1
+```
+
+### Summaries
+
+Extraction always generates **RAKE** (extractive keyword) summaries — instant, no LLM required. LLM-based summaries are opt-in via `--summarize` and require an Ollama or litellm-compatible model.
+
+RAKE summaries are used as the default for search and display. To add LLM summaries later:
+
+```bash
+acatome-extract enrich --summarize --skip-existing ~/.acatome/papers
+```
+
+### Sidecar metadata
+
+Place a `<stem>.meta.json` alongside any PDF to override metadata:
+
+```json
+{"type": "datasheet", "title": "LM317 Regulator", "author": "Texas Instruments", "year": 2022}
+```
+
+Supported fields: `type`, `title`, `author` (string or list), `year`, `doi`, `abstract`, `journal`.
+
+## Dependencies
+
+- **acatome-meta** — metadata lookup and verification
+- **marker-pdf** — structured PDF extraction
+- **litellm** / **Ollama** — LLM-based enrichment
+
+## Testing
+
+```bash
+uv run python -m pytest tests/ -v
+```
+
+## License
+
+GPL-3.0-or-later — see [LICENSE](LICENSE).

acatome_extract-0.2.0/README.md

@@ -0,0 +1,93 @@
+# acatome-extract
+
+PDF extraction and enrichment pipeline for scientific papers. Converts PDFs into structured, searchable bundles with block-level summaries and embeddings.
+
+## Features
+
+- **Marker PDF extraction** — structured block extraction with headings, tables, figures
+- **Fitz fallback** — recursive character chunking when Marker is unavailable
+- **LLM enrichment** — block and paper summaries via Ollama or litellm
+- **Embeddings** — sentence-transformer embeddings for semantic search
+- **File watcher** — `acatome-extract watch` monitors an inbox folder
+- **Bundle format** — `.acatome` companion files for sharing pre-built extractions
+- **CLI** — `acatome-extract` command for extract, enrich, and watch workflows
+
+## Installation
+
+```bash
+uv pip install -e .
+```
+
+With GPU acceleration:
+
+```bash
+uv pip install -e ".[gpu]"
+```
+
+## Usage
+
+```python
+from acatome_extract.pipeline import extract
+
+bundle = extract("/path/to/paper.pdf")
+```
+
+## CLI
+
+```bash
+# Extract (RAKE summaries included automatically, no LLM needed)
+acatome-extract extract paper.pdf
+acatome-extract extract --type datasheet TI_LM317.pdf   # non-article types
+
+# Enrich — embeddings only by default; add --summarize for LLM summaries
+acatome-extract enrich /path/to/bundle
+acatome-extract enrich --summarize /path/to/bundle       # enable LLM summaries
+acatome-extract enrich --summarize --skip-existing dir/   # incremental LLM pass
+
+# Watch — extract + embed + ingest; LLM summaries off by default
+acatome-extract watch ~/papers/inbox
+acatome-extract watch ~/papers/inbox --summarize          # enable LLM summaries
+
+# Migrate old bundles to new summaries dict format + add RAKE
+acatome-extract migrate ~/.acatome/papers
+acatome-extract migrate ~/.acatome/papers --dry-run       # preview changes
+
+# Supplements
+acatome-extract attach parent-slug supplement.pdf --name s1
+```
+
+### Summaries
+
+Extraction always generates **RAKE** (extractive keyword) summaries — instant, no LLM required. LLM-based summaries are opt-in via `--summarize` and require an Ollama or litellm-compatible model.
+
+RAKE summaries are used as the default for search and display. To add LLM summaries later:
+
+```bash
+acatome-extract enrich --summarize --skip-existing ~/.acatome/papers
+```
+
+### Sidecar metadata
+
+Place a `<stem>.meta.json` alongside any PDF to override metadata:
+
+```json
+{"type": "datasheet", "title": "LM317 Regulator", "author": "Texas Instruments", "year": 2022}
+```
+
+Supported fields: `type`, `title`, `author` (string or list), `year`, `doi`, `abstract`, `journal`.
+
+## Dependencies
+
+- **acatome-meta** — metadata lookup and verification
+- **marker-pdf** — structured PDF extraction
+- **litellm** / **Ollama** — LLM-based enrichment
+
+## Testing
+
+```bash
+uv run python -m pytest tests/ -v
+```
+
+## License
+
+GPL-3.0-or-later — see [LICENSE](LICENSE).

acatome_extract-0.2.0/pyproject.toml

@@ -0,0 +1,87 @@
+[project]
+name = "acatome-extract"
+version = "0.2.0"
+description = "PDF extraction pipeline for scientific papers"
+requires-python = ">=3.11"
+license = "GPL-3.0-or-later"
+authors = [{name = "Reto Stamm", email = "reto@retostamm.com"}]
+readme = "README.md"
+keywords = ["pdf", "extraction", "scientific-papers", "nlp", "summarization", "embeddings"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering",
+]
+
+dependencies = [
+    "acatome-meta>=0.1.0",
+    "marker-pdf>=1.0",
+    "precis-summary>=0.1.0",
+    "typer>=0.12",
+    "litellm>=1.40",
+    "watchdog>=4.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/acatome/acatome-extract"
+Repository = "https://github.com/acatome/acatome-extract"
+Issues = "https://github.com/acatome/acatome-extract/issues"
+
+[project.optional-dependencies]
+store = [
+    "acatome-store>=0.1.0",
+]
+embeddings = [
+    "sentence-transformers>=3.0",
+]
+gpu = [
+    "sentence-transformers>=3.0",
+    "torch>=2.0",
+]
+grobid = [
+    "grobid-client-python>=0.0.7",
+]
+dev = [
+    "pytest>=8.0",
+    "black>=24.0",
+    "ruff>=0.5",
+]
+
+[project.scripts]
+acatome-extract = "acatome_extract.cli:app"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/acatome_extract"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.black]
+line-length = 88
+
+[tool.ruff]
+line-length = 88
+
+[tool.bumpversion]
+current_version = "0.2.0"
+commit = true
+tag = true
+tag_name = "v{new_version}"
+
+[[tool.bumpversion.files]]
+filename = "pyproject.toml"
+search = 'version = "{current_version}"'
+replace = 'version = "{new_version}"'
+
+[[tool.bumpversion.files]]
+filename = "src/acatome_extract/__init__.py"
+search = '__version__ = "{current_version}"'
+replace = '__version__ = "{new_version}"'

acatome_extract-0.2.0/src/acatome_extract/__init__.py

@@ -0,0 +1,7 @@
+"""acatome-extract: PDF extraction pipeline for scientific papers."""
+
+from acatome_extract.bundle import read_bundle, write_bundle
+from acatome_extract.pipeline import extract, extract_dir
+
+__all__ = ["extract", "extract_dir", "read_bundle", "write_bundle"]
+__version__ = "0.2.0"

acatome_extract-0.2.0/src/acatome_extract/bundle.py

@@ -0,0 +1,55 @@
+"""Read/write .acatome bundle files (gzipped JSON)."""
+
+from __future__ import annotations
+
+import gzip
+import json
+from pathlib import Path
+from typing import Any
+
+
+def write_bundle(data: dict[str, Any], path: str | Path) -> Path:
+    """Write a bundle dict as gzipped JSON.
+
+    Args:
+        data: Bundle dict (header + blocks + enrichment_meta).
+        path: Output path (should end in .acatome).
+
+    Returns:
+        Path to written file.
+    """
+    path = Path(path)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with gzip.open(path, "wt", encoding="utf-8") as f:
+        json.dump(data, f, ensure_ascii=False, separators=(",", ":"))
+    return path
+
+
+def read_bundle(path: str | Path) -> dict[str, Any]:
+    """Read a .acatome bundle file.
+
+    Args:
+        path: Path to .acatome file.
+
+    Returns:
+        Parsed bundle dict.
+    """
+    path = Path(path)
+    with gzip.open(path, "rt", encoding="utf-8") as f:
+        return json.load(f)
+
+
+def update_bundle(
+    data: dict[str, Any],
+    path: str | Path,
+) -> Path:
+    """Write an already-modified bundle dict back to disk.
+
+    Args:
+        data: Bundle dict (header + blocks + enrichment_meta).
+        path: Path to .acatome file.
+
+    Returns:
+        Path to written file.
+    """
+    return write_bundle(data, path)

acatome_extract-0.2.0/src/acatome_extract/chunker.py

@@ -0,0 +1,152 @@
+"""Recursive character text splitter for document chunking.
+
+Splits text into chunks of roughly ``chunk_size`` characters, preferring
+to break at natural boundaries (paragraphs → newlines → sentences → words).
+Adjacent chunks overlap by ``chunk_overlap`` characters to preserve context
+across chunk boundaries.
+"""
+
+from __future__ import annotations
+
+import re
+
+# Default separators, tried in order (prefer paragraph → line → sentence → word)
+DEFAULT_SEPARATORS: list[str] = ["\n\n", "\n", ". ", ", ", " "]
+
+# Reasonable defaults for academic papers
+DEFAULT_CHUNK_SIZE = 800
+DEFAULT_CHUNK_OVERLAP = 150
+
+
+def split_text(
+    text: str,
+    chunk_size: int = DEFAULT_CHUNK_SIZE,
+    chunk_overlap: int = DEFAULT_CHUNK_OVERLAP,
+    separators: list[str] | None = None,
+) -> list[str]:
+    """Split *text* into chunks of approximately *chunk_size* characters.
+
+    The algorithm tries each separator in order.  For the first separator
+    that produces pieces, it keeps pieces that fit and recursively splits
+    those that don't (using the remaining separators).  Adjacent chunks
+    share *chunk_overlap* characters of context.
+
+    Returns a list of non-empty strings, each ≤ ``chunk_size`` chars
+    (unless a single word exceeds the limit, in which case it is kept
+    whole to avoid mid-word splits).
+    """
+    if not text.strip():
+        return []
+
+    if len(text) <= chunk_size:
+        return [text.strip()]
+
+    seps = separators if separators is not None else list(DEFAULT_SEPARATORS)
+
+    return _recursive_split(text, chunk_size, chunk_overlap, seps)
+
+
+def _recursive_split(
+    text: str,
+    chunk_size: int,
+    chunk_overlap: int,
+    separators: list[str],
+) -> list[str]:
+    """Core recursive splitting logic."""
+    # Base case: text fits
+    if len(text) <= chunk_size:
+        stripped = text.strip()
+        return [stripped] if stripped else []
+
+    # Try each separator
+    for i, sep in enumerate(separators):
+        pieces = _split_keeping_sep(text, sep)
+        if len(pieces) <= 1:
+            continue  # separator not found; try next
+
+        # Merge small pieces back together up to chunk_size
+        merged = _merge_pieces(pieces, chunk_size, chunk_overlap, sep)
+
+        # Recursively split any chunk that's still too big
+        remaining_seps = separators[i + 1 :]
+        result: list[str] = []
+        for chunk in merged:
+            if len(chunk) <= chunk_size:
+                stripped = chunk.strip()
+                if stripped:
+                    result.append(stripped)
+            elif remaining_seps:
+                result.extend(
+                    _recursive_split(chunk, chunk_size, chunk_overlap, remaining_seps)
+                )
+            else:
+                # No more separators — keep as-is (won't split mid-word)
+                stripped = chunk.strip()
+                if stripped:
+                    result.append(stripped)
+        return result
+
+    # No separator worked — return text as-is
+    stripped = text.strip()
+    return [stripped] if stripped else []
+
+
+def _split_keeping_sep(text: str, sep: str) -> list[str]:
+    """Split text by *sep*, keeping the separator at the start of each piece
+    (except the first)."""
+    parts = text.split(sep)
+    if len(parts) <= 1:
+        return parts
+
+    result = [parts[0]]
+    for part in parts[1:]:
+        result.append(sep + part)
+    return result
+
+
+def _merge_pieces(
+    pieces: list[str],
+    chunk_size: int,
+    chunk_overlap: int,
+    sep: str,
+) -> list[str]:
+    """Greedily merge adjacent pieces into chunks up to *chunk_size*.
+
+    When starting a new chunk, includes up to *chunk_overlap* characters
+    from the tail of the previous chunk.
+    """
+    chunks: list[str] = []
+    current: list[str] = []
+    current_len = 0
+
+    for piece in pieces:
+        piece_len = len(piece)
+
+        if current and current_len + piece_len > chunk_size:
+            # Flush current buffer
+            chunk_text = "".join(current)
+            if chunk_text.strip():
+                chunks.append(chunk_text)
+
+            # Build overlap from end of current buffer
+            overlap_pieces: list[str] = []
+            overlap_len = 0
+            for prev in reversed(current):
+                if overlap_len + len(prev) > chunk_overlap:
+                    break
+                overlap_pieces.insert(0, prev)
+                overlap_len += len(prev)
+
+            current = overlap_pieces
+            current_len = overlap_len
+
+        current.append(piece)
+        current_len += piece_len
+
+    # Flush remaining
+    if current:
+        chunk_text = "".join(current)
+        if chunk_text.strip():
+            chunks.append(chunk_text)
+
+    return chunks