paperflux 3.4.20260528__tar.gz

paperflux-3.4.20260528/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Emanuël Habets
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

paperflux-3.4.20260528/PKG-INFO

@@ -0,0 +1,93 @@
+Metadata-Version: 2.4
+Name: paperflux
+Version: 3.4.20260528
+Summary: AI-powered PDF annotation for research papers
+Author: PaperFlux contributors
+License-Expression: MIT
+Project-URL: Documentation, https://ehabets.github.io/PaperFlux/
+Project-URL: Homepage, https://github.com/ehabets/PaperFlux
+Project-URL: Issues, https://github.com/ehabets/PaperFlux/issues
+Project-URL: Source, https://github.com/ehabets/PaperFlux
+Keywords: annotation,openai,papers,pdf,research
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai>=2.38.0
+Requires-Dist: typer>=0.15.4
+Requires-Dist: PyMuPDF>=1.24
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: pydantic>=2.7
+Requires-Dist: rich>=13.7
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: jinja2>=3.1
+Provides-Extra: dev
+Requires-Dist: pytest>=8.2; extra == "dev"
+Dynamic: license-file
+
+# PaperFlux
+
+AI-powered PDF annotation for research papers. PaperFlux extracts exact quotations, organizes them by category (contributions, limitations, claims, evidence), and annotates your PDFs with precise highlights.
+
+## Quick Start
+
+### 1. Installation
+
+Install the latest release from PyPI:
+
+```bash
+python -m pip install paperflux
+```
+
+For local development from a cloned repository:
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+```
+
+### 2. Set API Key
+
+```bash
+export PAPERFLUX_OPENAI_API_KEY="sk-your-key"
+```
+
+### 3. Run
+
+```bash
+paperflux init
+paperflux --config config.yaml path/to/paper.pdf
+```
+
+## Features
+
+- Batch processing: `*.pdf`
+- Three detail levels (low/medium/high)
+- RAG-based extraction with exact quotes
+- Color-coded highlights by category
+- Markdown summary with sticky note
+- Quote-match report with matched/skipped counts and scores
+- Layout-aware quote matching across column, table, figure, and caption interruptions
+- Stage-level CLI progress during extraction and annotation
+- Configurable prompts and colors
+
+## Documentation
+
+For detailed setup, configuration options, and advanced usage, see the [full documentation](https://ehabets.github.io/PaperFlux/).
+
+## Contributing
+
+Contributions welcome! Fork the repo, create a feature branch, and open a PR.
+
+## License
+
+See [LICENSE](LICENSE) for details.

paperflux-3.4.20260528/README.md

@@ -0,0 +1,58 @@
+# PaperFlux
+
+AI-powered PDF annotation for research papers. PaperFlux extracts exact quotations, organizes them by category (contributions, limitations, claims, evidence), and annotates your PDFs with precise highlights.
+
+## Quick Start
+
+### 1. Installation
+
+Install the latest release from PyPI:
+
+```bash
+python -m pip install paperflux
+```
+
+For local development from a cloned repository:
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+```
+
+### 2. Set API Key
+
+```bash
+export PAPERFLUX_OPENAI_API_KEY="sk-your-key"
+```
+
+### 3. Run
+
+```bash
+paperflux init
+paperflux --config config.yaml path/to/paper.pdf
+```
+
+## Features
+
+- Batch processing: `*.pdf`
+- Three detail levels (low/medium/high)
+- RAG-based extraction with exact quotes
+- Color-coded highlights by category
+- Markdown summary with sticky note
+- Quote-match report with matched/skipped counts and scores
+- Layout-aware quote matching across column, table, figure, and caption interruptions
+- Stage-level CLI progress during extraction and annotation
+- Configurable prompts and colors
+
+## Documentation
+
+For detailed setup, configuration options, and advanced usage, see the [full documentation](https://ehabets.github.io/PaperFlux/).
+
+## Contributing
+
+Contributions welcome! Fork the repo, create a feature branch, and open a PR.
+
+## License
+
+See [LICENSE](LICENSE) for details.

paperflux-3.4.20260528/pyproject.toml

@@ -0,0 +1,65 @@
+[build-system]
+requires = ["setuptools>=77.0.3", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "paperflux"
+version = "3.4.20260528"
+description = "AI-powered PDF annotation for research papers"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [
+    { name = "PaperFlux contributors" },
+]
+keywords = ["annotation", "openai", "papers", "pdf", "research"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "Intended Audience :: Science/Research",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+]
+dependencies = [
+    "openai>=2.38.0",
+    "typer>=0.15.4",
+    "PyMuPDF>=1.24",
+    "pyyaml>=6.0",
+    "pydantic>=2.7",
+    "rich>=13.7",
+    "python-dotenv>=1.0.0",
+    "jinja2>=3.1",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.2",
+]
+
+[project.scripts]
+paperflux = "paperflux.cli:run"
+
+[project.urls]
+Documentation = "https://ehabets.github.io/PaperFlux/"
+Homepage = "https://github.com/ehabets/PaperFlux"
+Issues = "https://github.com/ehabets/PaperFlux/issues"
+Source = "https://github.com/ehabets/PaperFlux"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+paperflux = [
+    "templates/config.yaml",
+    "templates/prompts/*.j2",
+    "templates/prompts/*.txt",
+]
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]
+testpaths = ["tests"]

paperflux-3.4.20260528/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

paperflux-3.4.20260528/src/paperflux/__init__.py

@@ -0,0 +1 @@
+"""PaperFlux package."""

paperflux-3.4.20260528/src/paperflux/assistants.py

@@ -0,0 +1,363 @@
+import json
+import logging
+import time
+from pathlib import Path
+from typing import Any, Callable, Dict, Optional, Union
+
+from openai import OpenAI, AsyncOpenAI
+from .config import Config
+from jinja2 import Environment, FileSystemLoader
+
+logger = logging.getLogger(__name__)
+
+ProgressCallback = Callable[[str], None]
+
+
+def _resolve_config_path(path: Union[str, Path], cfg: Config) -> Path:
+    candidate = Path(path)
+    if candidate.is_absolute():
+        return candidate
+    config_dir = getattr(cfg, "_config_dir", None)
+    if config_dir is not None:
+        return config_dir / candidate
+    return candidate
+
+
+def _load_template(template_path: Union[str, Path]):
+    """
+    Load Jinja2 template given its filesystem path.
+    """
+    template_path = Path(template_path)
+    env = Environment(loader=FileSystemLoader(str(template_path.parent or Path("."))))
+    return env.get_template(template_path.name)
+
+
+def _load_text_file(path: Union[str, Path]) -> str:
+    with open(path, "r", encoding="utf-8") as handle:
+        return handle.read().strip()
+
+
+def _build_text_payload(format_payload: dict, verbosity: str) -> dict:
+    return {"format": format_payload, "verbosity": verbosity}
+
+
+def _multi_category_schema(max_quotes_per_category: int) -> dict:
+    schema = {
+        "type": "object",
+        "additionalProperties": False,
+        "properties": {
+            "categories": {
+                "type": "array",
+                "items": {
+                    "type": "object",
+                    "additionalProperties": False,
+                    "properties": {
+                        "name": {"type": "string"},
+                        "quotes": {
+                            "type": "array",
+                            "maxItems": max_quotes_per_category,
+                            "items": {
+                                "type": "object",
+                                "additionalProperties": False,
+                                "properties": {
+                                    "text": {"type": "string"},
+                                    "pages": {
+                                        "type": "array",
+                                        "items": {"type": "integer", "minimum": 1},
+                                    },
+                                    "prefix": {"type": "string"},
+                                    "suffix": {"type": "string"},
+                                },
+                                "required": ["text", "pages", "prefix", "suffix"],
+                            },
+                        },
+                        "category_summary": {"type": "string"},
+                    },
+                    "required": ["name", "quotes", "category_summary"],
+                },
+            }
+        },
+        "required": ["categories"],
+    }
+    return schema
+
+
+def _reasoning_payload(effort: str) -> Dict[str, str]:
+    return {"effort": effort}
+
+
+def _dump_failed_response(label: str, content: str) -> Optional[Path]:
+    """
+    Persist the raw response body when JSON parsing fails so users can inspect it.
+    """
+    try:
+        dump_dir = Path(".paperflux")
+        dump_dir.mkdir(exist_ok=True)
+        timestamp = int(time.time())
+        safe_label = "".join(c if c.isalnum() or c in ("-", "_") else "_" for c in label)[:40]
+        path = dump_dir / f"failed_response_{safe_label}_{timestamp}.txt"
+        path.write_text(content, encoding="utf-8")
+        return path
+    except Exception:
+        return None
+
+
+def _ensure_response_completed(resp: Any, context: str, max_output_tokens: int) -> None:
+    """
+    Raise a descriptive error if the Responses API indicates the run was truncated.
+    """
+    status = getattr(resp, "status", None)
+    if status and status != "completed":
+        incomplete = getattr(resp, "incomplete_details", None)
+        reason = getattr(incomplete, "reason", None)
+        raise ValueError(
+            f"{context} response ended with status='{status}' (reason='{reason}'). "
+            f"Consider increasing ui.max_output_tokens (currently {max_output_tokens}) "
+            "or lowering ui.reasoning_effort, detail_level, category count, or "
+            "rag.max_quotes_per_category."
+        )
+
+
+def _extract_response_text(resp) -> str:
+    """
+    Best-effort extraction of plain text from Responses API result.
+    Prefers resp.output_text when available, with fallbacks for structured outputs.
+    """
+    # Newer SDKs expose output_text
+    text = getattr(resp, "output_text", None)
+    if isinstance(text, str) and text.strip():
+        return text
+
+    # Fallback: iterate output structure
+    try:
+        output = getattr(resp, "output", None)
+        if output and isinstance(output, list):
+            chunks = []
+            for item in output:
+                # message/content/output_text structure
+                if getattr(item, "type", None) == "message":
+                    for c in getattr(item, "content", []) or []:
+                        if getattr(c, "type", None) == "output_text":
+                            t = getattr(c, "text", "")
+                            if t:
+                                chunks.append(t)
+            if chunks:
+                return "".join(chunks)
+    except (AttributeError, TypeError, ValueError):  # fallback for unknown SDK shapes
+        pass
+
+    # Last resort: stringify
+    return str(resp)
+
+
+def _extract_parsed_json(resp: Any) -> Optional[dict]:
+    """
+    Try to extract structured output when response_format=json_schema is used.
+    """
+    parsed = getattr(resp, "output_parsed", None)
+    if parsed:
+        if isinstance(parsed, list) and parsed and isinstance(parsed[0], dict):
+            return parsed[0]
+        if isinstance(parsed, dict):
+            return parsed
+    output = getattr(resp, "output", None)
+    if isinstance(output, list):
+        for item in output:
+            for c in getattr(item, "content", []) or []:
+                maybe = getattr(c, "parsed", None)
+                if isinstance(maybe, dict):
+                    return maybe
+    return None
+
+
+async def analyze_pdf(
+    path: Path,
+    cfg: Config,
+    progress_callback: Optional[ProgressCallback] = None,
+) -> dict:
+    """
+    Use OpenAI Responses API with built-in file_search tool to retrieve per-category evidence
+    and synthesize a global summary. Retrieval happens server-side with tool_choice='auto'.
+    - Creates a vector store
+    - Uploads the PDF
+    - Runs one bundled category extraction response with file_search enabled.
+    - Generates a final markdown summary from category summaries.
+    """
+    # Sync client used for vector store creation/upload (simpler surface)
+    client_sync = OpenAI(api_key=cfg.openai.api_key)
+    vector_store_id: Optional[str] = None
+
+    try:
+        # 1) Create vector store and upload PDF
+        if progress_callback:
+            progress_callback("Creating temporary vector store")
+        vector_store = client_sync.vector_stores.create(
+            name="PaperFlux Vector Store",
+            expires_after={
+                "anchor": "last_active_at",
+                "days": cfg.rag.vector_store_expires_after_days,
+            },
+        )
+        vector_store_id = vector_store.id
+        if progress_callback:
+            progress_callback(f"Uploading and indexing {path.name}")
+        with open(path, "rb") as f:
+            client_sync.vector_stores.files.upload_and_poll(
+                vector_store_id=vector_store_id,
+                file=f
+            )
+
+        categories = cfg.extraction_categories.categories  # Dict[name, description]
+
+        # 2) Bundle categories into a single Responses job with file_search tool
+        client_async = AsyncOpenAI(api_key=cfg.openai.api_key)
+        category_template = _load_template(_resolve_config_path(cfg.rag.category_prompt_file, cfg))
+        category_system_prompt = _load_text_file(
+            _resolve_config_path(cfg.rag.category_system_prompt_file, cfg)
+        )
+
+        category_entries = [
+            {"name": cat, "description": desc}
+            for cat, desc in categories.items()
+        ]
+        user_msg = category_template.render(
+            categories=category_entries,
+            max_quotes_per_category=cfg.rag.max_quotes_per_category,
+        )
+
+        file_search_tool = {
+            "type": "file_search",
+            "vector_store_ids": [vector_store_id],
+        }
+        if cfg.rag.max_num_results is not None:
+            file_search_tool["max_num_results"] = cfg.rag.max_num_results
+        tools = [file_search_tool]
+
+        schema = _multi_category_schema(cfg.rag.max_quotes_per_category)
+        text_payload = _build_text_payload(
+            {
+                "type": "json_schema",
+                "name": "multi_category_schema",
+                "schema": schema,
+                "strict": True,
+            },
+            cfg.ui.verbosity,
+        )
+        reasoning_payload = _reasoning_payload(cfg.ui.reasoning_effort)
+        request_input = [
+            {"role": "system", "content": category_system_prompt},
+            {"role": "user", "content": user_msg},
+        ]
+        kwargs = {
+            "model": cfg.openai.model,
+            "input": request_input,
+            "max_output_tokens": cfg.ui.max_output_tokens,
+            "tools": tools,
+            "text": text_payload,
+            "tool_choice": {"type": "file_search"},
+            "store": False,
+        }
+        if cfg.rag.include_search_results:
+            kwargs["include"] = ["file_search_call.results"]
+        kwargs["reasoning"] = reasoning_payload
+        if progress_callback:
+            progress_callback("Extracting quotes with OpenAI")
+        multi_resp = await client_async.responses.create(**kwargs)
+        _ensure_response_completed(multi_resp, "Category bundle", cfg.ui.max_output_tokens)
+
+        result = _extract_parsed_json(multi_resp)
+        if result is None:
+            text_val = _extract_response_text(multi_resp)
+            if not text_val or not text_val.strip():
+                dump_path = _dump_failed_response(
+                    "category_bundle_no_parsed_json",
+                    getattr(multi_resp, "output_text", "") or str(multi_resp),
+                )
+                hint = f" (raw saved to {dump_path})" if dump_path else ""
+                raise ValueError(f"Category bundle response missing parsed JSON{hint}")
+            result = json.loads(text_val)
+        if not isinstance(result, dict):
+            raise ValueError(f"Category bundle returned non-dict JSON: {type(result)}")
+
+        bundle_list = result.get("categories")
+        if not isinstance(bundle_list, list):
+            raise ValueError("Category bundle JSON missing 'categories' array")
+
+        quotes: Dict[str, list] = {}
+        category_summaries: Dict[str, str] = {}
+        for entry in bundle_list:
+            if not isinstance(entry, dict):
+                continue
+            category = entry.get("name") or entry.get("category")
+            if not isinstance(category, str):
+                continue
+            cat_data = entry
+            raw_quotes = (
+                cat_data.get("quotes")
+                or cat_data.get("evidence")
+                or cat_data.get("quote_list")
+                or []
+            )
+            normalised_quotes = []
+            for item in raw_quotes:
+                if isinstance(item, dict):
+                    text_val = item.get("text") or item.get("quote") or item.get("content")
+                    if not isinstance(text_val, str) or not text_val.strip():
+                        continue
+                    pages_val = item.get("pages") or item.get("page")
+                    if isinstance(pages_val, int):
+                        pages = [pages_val]
+                    elif isinstance(pages_val, list):
+                        pages = [p for p in pages_val if isinstance(p, int) and p > 0]
+                    else:
+                        pages = []
+                    prefix_val = item.get("prefix") or item.get("context_before") or ""
+                    suffix_val = item.get("suffix") or item.get("context_after") or ""
+                    normalised_quotes.append({
+                        "text": text_val.strip(),
+                        "pages": pages,
+                        "prefix": prefix_val.strip() if isinstance(prefix_val, str) else "",
+                        "suffix": suffix_val.strip() if isinstance(suffix_val, str) else "",
+                    })
+                elif isinstance(item, str) and item.strip():
+                    normalised_quotes.append({
+                        "text": item.strip(),
+                        "pages": [],
+                        "prefix": "",
+                        "suffix": "",
+                    })
+            quotes[category] = normalised_quotes
+            summary_val = cat_data.get("category_summary") or cat_data.get("summary") or ""
+            category_summaries[category] = summary_val
+
+        # 3) Synthesize global summary (no tools required)
+        summary_template = _load_template(_resolve_config_path(cfg.rag.summary_prompt_file, cfg))
+        summary_msg = summary_template.render(
+            detail_level=cfg.ui.detail_level,
+            category_summaries=category_summaries
+        )
+        summary_text_payload = _build_text_payload({"type": "text"}, cfg.ui.verbosity)
+        summary_reasoning = _reasoning_payload(cfg.ui.reasoning_effort)
+        summary_kwargs = {
+            "model": cfg.openai.model,
+            "input": summary_msg,
+            "max_output_tokens": cfg.ui.max_output_tokens,
+            "text": summary_text_payload,
+            "store": False,
+        }
+        summary_kwargs["reasoning"] = summary_reasoning
+        if progress_callback:
+            progress_callback("Generating summary")
+        summary_resp = await client_async.responses.create(**summary_kwargs)
+        _ensure_response_completed(summary_resp, "Summary", cfg.ui.max_output_tokens)
+        key_takeaways = _extract_response_text(summary_resp)
+
+        return {"key_takeaways": key_takeaways, "quotes": quotes}
+    finally:
+        if vector_store_id:
+            try:
+                if progress_callback:
+                    progress_callback("Cleaning up temporary vector store")
+                client_sync.vector_stores.delete(vector_store_id=vector_store_id)
+            except Exception as exc:
+                logger.warning("Failed to delete vector store %s: %s", vector_store_id, exc)