openalex-local 0.1.0__py3-none-any.whl → 0.3.1__py3-none-any.whl

openalex_local-0.3.1.dist-info/METADATA

@@ -0,0 +1,288 @@
+Metadata-Version: 2.4
+Name: openalex-local
+Version: 0.3.1
+Summary: Local OpenAlex database with 284M+ works, abstracts, and semantic search
+Author-email: Yusuke Watanabe <ywatanabe@alumni.u-tokyo.ac.jp>
+License: AGPL-3.0
+Project-URL: Homepage, https://github.com/ywatanabe1989/openalex-local
+Project-URL: Repository, https://github.com/ywatanabe1989/openalex-local
+Keywords: openalex,academic,research,abstracts,semantic-search
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: GNU Affero General Public License v3
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: click>=8.0
+Requires-Dist: awscli>=1.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21; extra == "dev"
+Provides-Extra: mcp
+Requires-Dist: fastmcp>=0.4; extra == "mcp"
+Provides-Extra: server
+Requires-Dist: fastapi>=0.100; extra == "server"
+Requires-Dist: uvicorn>=0.23; extra == "server"
+Provides-Extra: docs
+Requires-Dist: sphinx>=7.0; extra == "docs"
+Requires-Dist: sphinx-rtd-theme>=2.0; extra == "docs"
+Requires-Dist: myst-parser>=2.0; extra == "docs"
+Requires-Dist: sphinx-copybutton>=0.5; extra == "docs"
+Requires-Dist: sphinx-autodoc-typehints>=1.25; extra == "docs"
+Provides-Extra: all
+Requires-Dist: openalex-local[dev,docs,mcp,server]; extra == "all"
+
+# OpenAlex Local
+
+Local OpenAlex database with 284M+ scholarly works, abstracts, and semantic search.
+
+[![PyPI version](https://badge.fury.io/py/openalex-local.svg)](https://badge.fury.io/py/openalex-local)
+[![Documentation](https://readthedocs.org/projects/openalex-local/badge/?version=latest)](https://openalex-local.readthedocs.io/en/latest/)
+[![Tests](https://github.com/ywatanabe1989/openalex-local/actions/workflows/test.yml/badge.svg)](https://github.com/ywatanabe1989/openalex-local/actions/workflows/test.yml)
+[![Python](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/badge/license-AGPL--3.0-blue.svg)](LICENSE)
+
+<details>
+<summary><strong>Why OpenAlex Local?</strong></summary>
+
+**Built for the LLM era** - features that matter for AI research assistants:
+
+| Feature | Benefit |
+|---------|---------|
+| **284M Works** | More coverage than CrossRef |
+| **Abstracts** | ~45-60% availability for semantic search |
+| **Concepts & Topics** | Built-in classification |
+| **Author Disambiguation** | Linked to institutions |
+| **Open Access Info** | OA status and URLs |
+
+Perfect for: RAG systems, research assistants, literature review automation.
+
+</details>
+
+<details>
+<summary><strong>Installation</strong></summary>
+
+```bash
+pip install openalex-local
+```
+
+From source:
+```bash
+git clone https://github.com/ywatanabe1989/openalex-local
+cd openalex-local && make install
+```
+
+Database setup (~300 GB, ~1-2 days to build):
+```bash
+# Check system status
+make status
+
+# 1. Download OpenAlex Works snapshot (~300GB)
+make download-screen  # runs in background
+
+# 2. Build SQLite database
+make build-db
+
+# 3. Build FTS5 index
+make build-fts
+```
+
+</details>
+
+<details>
+<summary><strong>Python API</strong></summary>
+
+```python
+from openalex_local import search, get, count
+
+# Full-text search (title + abstract)
+results = search("machine learning neural networks")
+for work in results:
+    print(f"{work.title} ({work.year})")
+    print(f"  Abstract: {work.abstract[:200]}...")
+    print(f"  Concepts: {[c['name'] for c in work.concepts]}")
+
+# Get by OpenAlex ID or DOI
+work = get("W2741809807")
+work = get("10.1038/nature12373")
+
+# Count matches
+n = count("CRISPR")
+```
+
+</details>
+
+<details>
+<summary><strong>CLI</strong></summary>
+
+```bash
+openalex-local search "CRISPR genome editing" -n 5
+openalex-local search-by-doi W2741809807
+openalex-local search-by-doi 10.1038/nature12373
+openalex-local status  # Configuration and database stats
+```
+
+With abstracts (`-a` flag):
+```
+$ openalex-local search "neural network" -n 1 -a
+
+Found 1,523,847 matches in 45.2ms
+
+1. Deep learning for neural networks (2015)
+   OpenAlex ID: W2741809807
+   Abstract: This paper presents a comprehensive overview of deep learning
+   techniques for neural network architectures...
+```
+
+</details>
+
+<details>
+<summary><strong>HTTP API</strong></summary>
+
+Start the FastAPI server:
+```bash
+openalex-local relay --host 0.0.0.0 --port 31292
+```
+
+Endpoints:
+```bash
+# Search works (FTS5)
+curl "http://localhost:31292/works?q=CRISPR&limit=10"
+
+# Get by ID or DOI
+curl "http://localhost:31292/works/W2741809807"
+curl "http://localhost:31292/works/10.1038/nature12373"
+
+# Batch lookup
+curl -X POST "http://localhost:31292/works/batch" \
+  -H "Content-Type: application/json" \
+  -d '{"ids": ["W2741809807", "10.1038/nature12373"]}'
+
+# Database info
+curl "http://localhost:31292/info"
+```
+
+HTTP mode (connect to running server):
+```bash
+# On local machine (if server is remote)
+ssh -L 31292:127.0.0.1:31292 your-server
+
+# Python client
+from openalex_local import configure_http
+configure_http("http://localhost:31292")
+
+# Or via CLI
+openalex-local --http search "CRISPR"
+```
+
+</details>
+
+<details>
+<summary><strong>MCP Server</strong></summary>
+
+Run as MCP (Model Context Protocol) server:
+```bash
+openalex-local mcp start
+```
+
+Local MCP client configuration:
+```json
+{
+  "mcpServers": {
+    "openalex-local": {
+      "command": "openalex-local",
+      "args": ["mcp", "start"],
+      "env": {
+        "OPENALEX_LOCAL_DB": "/path/to/openalex.db"
+      }
+    }
+  }
+}
+```
+
+Remote MCP via HTTP:
+```bash
+# On server: start persistent MCP server
+openalex-local mcp start -t http --host 0.0.0.0 --port 8083
+```
+```json
+{
+  "mcpServers": {
+    "openalex-remote": {
+      "url": "http://your-server:8083/mcp"
+    }
+  }
+}
+```
+
+Diagnose setup:
+```bash
+openalex-local mcp doctor        # Check dependencies and database
+openalex-local mcp list-tools    # Show available MCP tools
+openalex-local mcp installation  # Show client config examples
+```
+
+Available tools:
+- `search` - Full-text search across 284M+ papers
+- `search_by_id` - Get paper by OpenAlex ID or DOI
+- `enrich_ids` - Batch lookup with metadata
+- `status` - Database statistics
+
+</details>
+
+<details>
+<summary><strong>Related Projects</strong></summary>
+
+**[crossref-local](https://github.com/ywatanabe1989/crossref-local)** - Sister project with CrossRef data:
+
+| Feature | crossref-local | openalex-local |
+|---------|----------------|----------------|
+| Works | 167M | 284M |
+| Abstracts | ~21% | ~45-60% |
+| Update frequency | Real-time | Monthly |
+| DOI authority | Yes (source) | Uses CrossRef |
+| Citations | Raw references | Linked works |
+| Concepts/Topics | No | Yes |
+| Author IDs | No | Yes |
+| Best for | DOI lookup, raw refs | Semantic search |
+
+**When to use CrossRef**: Real-time DOI updates, raw reference parsing, authoritative metadata.
+**When to use OpenAlex**: Semantic search, citation analysis, topic discovery.
+
+</details>
+
+<details>
+<summary><strong>Documentation</strong></summary>
+
+Full documentation available at [openalex-local.readthedocs.io](https://openalex-local.readthedocs.io/en/latest/)
+
+- [Installation Guide](https://openalex-local.readthedocs.io/en/latest/installation.html)
+- [Quickstart](https://openalex-local.readthedocs.io/en/latest/quickstart.html)
+- [CLI Reference](https://openalex-local.readthedocs.io/en/latest/cli_reference.html)
+- [HTTP API Reference](https://openalex-local.readthedocs.io/en/latest/http_api.html)
+- [Python API](https://openalex-local.readthedocs.io/en/latest/api/openalex_local.html)
+
+</details>
+
+<details>
+<summary><strong>Data Source</strong></summary>
+
+Data from [OpenAlex](https://openalex.org/), an open catalog of scholarly works.
+Updated monthly from their [snapshot](https://docs.openalex.org/download-all-data/openalex-snapshot).
+
+</details>
+
+---
+
+<p align="center">
+  <a href="https://scitex.ai"><img src="docs/scitex-icon-navy-inverted.png" alt="SciTeX" width="40"/></a>
+  <br>
+  AGPL-3.0 · ywatanabe@scitex.ai
+</p>
+
+<!-- EOF -->

openalex_local-0.3.1.dist-info/RECORD

@@ -0,0 +1,34 @@
+openalex_local/__init__.py,sha256=USWssgVhl3PXKugOWe_VNJRD0j_PR-0vrXziWeaYX6k,1057
+openalex_local/__main__.py,sha256=7zIPyOv659VptzHef3Zsw3k6m-WhGTN4MFq2-yVkdLE,111
+openalex_local/aio.py,sha256=42qi3qOKgaK-e37rVj9afGvzLhzDhHAuBVOqtmXEh6M,6468
+openalex_local/cache.py,sha256=-FdYvzd9XSM9qbuE2xHu8p-BUjF6pItv1qX7Jl0auck,415
+openalex_local/cli.py,sha256=2pWJK_vnO1IIwUrMB3K8KW94ZWfkEF7TLSxMsaR-LrI,137
+openalex_local/jobs.py,sha256=8yoG1um3g94wddPcnhx5w3XDPegnG7qpRshDtCcR2gI,4892
+openalex_local/remote.py,sha256=PmvUq87mC76sM9BL9RczOaXtvwHoqMc_dN5PJCYT18M,239
+openalex_local/server.py,sha256=SKoQ-cOoZjdXm24Sv1CFu3F8UclbD6QstfQb-7l2xtA,215
+openalex_local/_cache/__init__.py,sha256=z56OFC31_zmngyD7k7N8Tt0vFqjIgYN7-crhmPSeG98,891
+openalex_local/_cache/core.py,sha256=fi8lSVhnmhSSLJ4rEJIKMZWALTat4NlXuyZxu2awBvw,8277
+openalex_local/_cache/export.py,sha256=uHpymppj8n-zz2KrveHH0XO9Mo1wR81eQ2B8CWsM6nw,2713
+openalex_local/_cache/models.py,sha256=llxMa2gB8cVsmhz2Lt8YlpFG2yTtIbv6OO1O8lw-Aso,319
+openalex_local/_cache/utils.py,sha256=xPQpUudWU1y-KuzKnNQN5Rt0R-fnEqcno7_EFrpgINY,2038
+openalex_local/_cli/__init__.py,sha256=NS07Eo93dRAuO5vxGwGwS0ctvZxMMkF_py09bzIk3Hk,175
+openalex_local/_cli/cli.py,sha256=S3RK3BzFBFWdwFPoiiImgZwMFhkRcNjK74RVP_Vp-vM,12874
+openalex_local/_cli/cli_cache.py,sha256=cy1LHBtJ7S7ecLgbsCFisH9XnQ6gDm0o1vT2QAUrBOU,7147
+openalex_local/_cli/mcp.py,sha256=IZ5r3rs601eTACkFhxojzOxJh9QJHLSxTjzUAilnlDs,6306
+openalex_local/_cli/mcp_server.py,sha256=5qMWxb1Yeqf4GEH8JIPKRrM4lb-tOJlKHPbMjJxWGlY,6829
+openalex_local/_core/__init__.py,sha256=aj7jUJ8Kv9cWoLC120JGF6h_kmgf2-5wwHkkTsgqwGU,615
+openalex_local/_core/api.py,sha256=_C6hWhmvRjqpdn_LEzLhm_krbkE9YIGY3C-tep0yN20,9410
+openalex_local/_core/config.py,sha256=VEoFIVCuFLO2zPRpJ0aeG5Mu8nqtHN2eFOQeGxFdQ7Y,3372
+openalex_local/_core/db.py,sha256=wyCNRbNxNujfvdJwvnNjrzhJGC5aS2ZhoKaSuuheehs,5957
+openalex_local/_core/export.py,sha256=NBt4dldvqPus7Ns88Qi9yPd4KgMX5FdggsIIZMfBWM4,6517
+openalex_local/_core/fts.py,sha256=52TrRqabNj1zLR8gDtb9mOgIXioEBaTRJN7tz5mAcpE,3959
+openalex_local/_core/models.py,sha256=m4ApslrAuTsCz8j_0Zfc6CXVWweBU8VCeBiOrtvH3Ng,13575
+openalex_local/_remote/__init__.py,sha256=3NmHe2uY_VDR-MxP2XAZM8EW9PYQ57_s77hZFVGILms,792
+openalex_local/_remote/base.py,sha256=qzyD-D_JPX9g5O9bJt01JYf2fzEC-7A7uW5mUA_qRIo,8404
+openalex_local/_server/__init__.py,sha256=8mZawWpAuGFwNJaXekrtaFgrb-1wcXaHiIzB0pVmFug,2785
+openalex_local/_server/routes.py,sha256=3qTcjiP2huLVrB6dHjCTrKossRGNFacHCJX3Ai6u4K4,4444
+openalex_local-0.3.1.dist-info/METADATA,sha256=T0CwMb4_pEW3Ol8zeiynMNGMaqHCDFcDx7m2PtTY4fA,8294
+openalex_local-0.3.1.dist-info/WHEEL,sha256=wUyA8OaulRlbfwMtmQsvNngGrxQHAvkKcvRmdizlJi0,92
+openalex_local-0.3.1.dist-info/entry_points.txt,sha256=8G5Q3Nwg3vlKqJiHO0BoAgbSSi3R4lNBMqwANbs7Uz4,64
+openalex_local-0.3.1.dist-info/top_level.txt,sha256=arEhuDR1f42p7soJ5JkJFAnAiCPSGKvuHIchaKTmqKg,15
+openalex_local-0.3.1.dist-info/RECORD,,

{openalex_local-0.1.0.dist-info → openalex_local-0.3.1.dist-info}/WHEEL

@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: setuptools (80.9.0)
+Generator: setuptools (80.10.2)
 Root-Is-Purelib: true
 Tag: py3-none-any
 

openalex_local-0.3.1.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+openalex-local = openalex_local._cli.cli:main

openalex_local/config.py

@@ -1,73 +0,0 @@
-"""Configuration for openalex_local."""
-
-import os
-from pathlib import Path
-from typing import Optional
-
-# Default database locations (checked in order)
-DEFAULT_DB_PATHS = [
-    Path("/home/ywatanabe/proj/openalex-local/data/openalex.db"),
-    Path("/home/ywatanabe/proj/openalex_local/data/openalex.db"),
-    Path("/mnt/nas_ug/openalex_local/data/openalex.db"),
-    Path.home() / ".openalex_local" / "openalex.db",
-    Path.cwd() / "data" / "openalex.db",
-]
-
-
-def get_db_path() -> Path:
-    """
-    Get database path from environment or auto-detect.
-
-    Priority:
-    1. OPENALEX_LOCAL_DB environment variable
-    2. First existing path from DEFAULT_DB_PATHS
-
-    Returns:
-        Path to the database file
-
-    Raises:
-        FileNotFoundError: If no database found
-    """
-    # Check environment variable first
-    env_path = os.environ.get("OPENALEX_LOCAL_DB")
-    if env_path:
-        path = Path(env_path)
-        if path.exists():
-            return path
-        raise FileNotFoundError(f"OPENALEX_LOCAL_DB path not found: {env_path}")
-
-    # Auto-detect from default locations
-    for path in DEFAULT_DB_PATHS:
-        if path.exists():
-            return path
-
-    raise FileNotFoundError(
-        "OpenAlex database not found. Set OPENALEX_LOCAL_DB environment variable "
-        f"or place database at one of: {[str(p) for p in DEFAULT_DB_PATHS]}"
-    )
-
-
-class Config:
-    """Configuration container."""
-
-    _db_path: Optional[Path] = None
-
-    @classmethod
-    def get_db_path(cls) -> Path:
-        """Get or auto-detect database path."""
-        if cls._db_path is None:
-            cls._db_path = get_db_path()
-        return cls._db_path
-
-    @classmethod
-    def set_db_path(cls, path: str | Path) -> None:
-        """Set database path explicitly."""
-        path = Path(path)
-        if not path.exists():
-            raise FileNotFoundError(f"Database not found: {path}")
-        cls._db_path = path
-
-    @classmethod
-    def reset(cls) -> None:
-        """Reset configuration (for testing)."""
-        cls._db_path = None

openalex_local/models.py

@@ -1,187 +0,0 @@
-"""Data models for openalex_local."""
-
-from dataclasses import dataclass, field
-from typing import List, Optional, Dict, Any
-
-
-@dataclass
-class Work:
-    """
-    Represents a scholarly work from OpenAlex.
-
-    Attributes:
-        openalex_id: OpenAlex ID (e.g., W2741809807)
-        doi: Digital Object Identifier
-        title: Work title
-        abstract: Abstract text (reconstructed from inverted index)
-        authors: List of author names
-        year: Publication year
-        source: Journal/venue name
-        issn: Journal ISSN
-        volume: Volume number
-        issue: Issue number
-        pages: Page range
-        publisher: Publisher name
-        type: Work type (journal-article, book-chapter, etc.)
-        concepts: List of OpenAlex concepts
-        topics: List of OpenAlex topics
-        cited_by_count: Number of citations
-        referenced_works: List of referenced OpenAlex IDs
-        is_oa: Is open access
-        oa_url: Open access URL
-    """
-
-    openalex_id: str
-    doi: Optional[str] = None
-    title: Optional[str] = None
-    abstract: Optional[str] = None
-    authors: List[str] = field(default_factory=list)
-    year: Optional[int] = None
-    source: Optional[str] = None
-    issn: Optional[str] = None
-    volume: Optional[str] = None
-    issue: Optional[str] = None
-    pages: Optional[str] = None
-    publisher: Optional[str] = None
-    type: Optional[str] = None
-    concepts: List[Dict[str, Any]] = field(default_factory=list)
-    topics: List[Dict[str, Any]] = field(default_factory=list)
-    cited_by_count: Optional[int] = None
-    referenced_works: List[str] = field(default_factory=list)
-    is_oa: bool = False
-    oa_url: Optional[str] = None
-
-    @classmethod
-    def from_openalex(cls, data: dict) -> "Work":
-        """
-        Create Work from OpenAlex API/snapshot JSON.
-
-        Args:
-            data: OpenAlex work dictionary
-
-        Returns:
-            Work instance
-        """
-        # Extract OpenAlex ID
-        openalex_id = data.get("id", "").replace("https://openalex.org/", "")
-
-        # Extract DOI
-        doi = data.get("doi", "").replace("https://doi.org/", "") if data.get("doi") else None
-
-        # Extract authors
-        authors = []
-        for authorship in data.get("authorships", []):
-            author = authorship.get("author", {})
-            name = author.get("display_name")
-            if name:
-                authors.append(name)
-
-        # Reconstruct abstract from inverted index
-        abstract = None
-        inv_index = data.get("abstract_inverted_index")
-        if inv_index:
-            words = sorted(
-                [(pos, word) for word, positions in inv_index.items() for pos in positions]
-            )
-            abstract = " ".join(word for _, word in words)
-
-        # Extract source info
-        primary_location = data.get("primary_location") or {}
-        source_info = primary_location.get("source") or {}
-        source = source_info.get("display_name")
-        issns = source_info.get("issn") or []
-        issn = issns[0] if issns else None
-
-        # Extract biblio
-        biblio = data.get("biblio") or {}
-
-        # Extract concepts (top 5)
-        concepts = [
-            {"name": c.get("display_name"), "score": c.get("score")}
-            for c in (data.get("concepts") or [])[:5]
-        ]
-
-        # Extract topics (top 3)
-        topics = [
-            {"name": t.get("display_name"), "subfield": t.get("subfield", {}).get("display_name")}
-            for t in (data.get("topics") or [])[:3]
-        ]
-
-        # Extract OA info
-        oa_info = data.get("open_access") or {}
-
-        return cls(
-            openalex_id=openalex_id,
-            doi=doi,
-            title=data.get("title") or data.get("display_name"),
-            abstract=abstract,
-            authors=authors,
-            year=data.get("publication_year"),
-            source=source,
-            issn=issn,
-            volume=biblio.get("volume"),
-            issue=biblio.get("issue"),
-            pages=biblio.get("first_page"),
-            publisher=source_info.get("host_organization_name"),
-            type=data.get("type"),
-            concepts=concepts,
-            topics=topics,
-            cited_by_count=data.get("cited_by_count"),
-            referenced_works=[
-                r.replace("https://openalex.org/", "")
-                for r in (data.get("referenced_works") or [])
-            ],
-            is_oa=oa_info.get("is_oa", False),
-            oa_url=oa_info.get("oa_url"),
-        )
-
-    def to_dict(self) -> dict:
-        """Convert to dictionary."""
-        return {
-            "openalex_id": self.openalex_id,
-            "doi": self.doi,
-            "title": self.title,
-            "abstract": self.abstract,
-            "authors": self.authors,
-            "year": self.year,
-            "source": self.source,
-            "issn": self.issn,
-            "volume": self.volume,
-            "issue": self.issue,
-            "pages": self.pages,
-            "publisher": self.publisher,
-            "type": self.type,
-            "concepts": self.concepts,
-            "topics": self.topics,
-            "cited_by_count": self.cited_by_count,
-            "referenced_works": self.referenced_works,
-            "is_oa": self.is_oa,
-            "oa_url": self.oa_url,
-        }
-
-
-@dataclass
-class SearchResult:
-    """
-    Container for search results with metadata.
-
-    Attributes:
-        works: List of Work objects
-        total: Total number of matches
-        query: Original search query
-        elapsed_ms: Search time in milliseconds
-    """
-
-    works: List[Work]
-    total: int
-    query: str
-    elapsed_ms: float
-
-    def __len__(self) -> int:
-        return len(self.works)
-
-    def __iter__(self):
-        return iter(self.works)
-
-    def __getitem__(self, idx):
-        return self.works[idx]