preprint-fulltext 0.1.0__tar.gz

preprint_fulltext-0.1.0/.claude-plugin/marketplace.json

@@ -0,0 +1,15 @@
+{
+  "name": "preprint-fulltext",
+  "owner": {
+    "name": "Min Dai",
+    "email": "dai@broadinstitute.org",
+    "url": "https://fishelllab.hms.harvard.edu"
+  },
+  "plugins": [
+    {
+      "name": "preprint-fulltext",
+      "source": ".",
+      "description": "Retrieve bioRxiv/medRxiv/arXiv preprint full text as structured sections; search/discover preprints; build embedding-ready corpora. CLI + MCP."
+    }
+  ]
+}

preprint_fulltext-0.1.0/.claude-plugin/plugin.json

@@ -0,0 +1,12 @@
+{
+  "name": "preprint-fulltext",
+  "version": "0.1.0",
+  "description": "Retrieve full text of bioRxiv/medRxiv/arXiv preprints as structured sections to ground scientific reasoning and deep research, search/discover preprints, and build embedding-ready corpora (Europe PMC + openRxiv S3 + arXiv LaTeXML), via CLI and MCP.",
+  "author": {
+    "name": "Min Dai",
+    "email": "dai@broadinstitute.org",
+    "url": "https://fishelllab.hms.harvard.edu"
+  },
+  "homepage": "https://github.com/genecell/preprint-fulltext",
+  "license": "BSD-3-Clause"
+}

preprint_fulltext-0.1.0/.cursor/rules/preprint-fulltext.mdc

@@ -0,0 +1,106 @@
+---
+description: Retrieve full text of bioRxiv/medRxiv/arXiv preprints as structured sections to ground scientific reasoning and deep research (methods/results, not just abstracts), or build embedding-ready corpora. Use when a task needs a preprint's full text or sections from a DOI/arXiv id/URL; to verify a claim against a paper's Results; to extract methods or parameters; keyword/title/author search over preprints; discovery by topic/date; or a chunked JSONL/Parquet corpus for embeddings/RAG. Triggers on: bioRxiv, medRxiv, arXiv, openRxiv, preprint full text, JATS, a 10.1101/ or 10.64898/ DOI, an arXiv id (2401.10515), "get the methods/results section", "what does this paper actually do", "search preprints".
+globs:
+alwaysApply: false
+---
+
+# preprint-fulltext
+
+A CLI + Python library + MCP server for retrieving bioRxiv/medRxiv preprint full text
+and turning it into structured, embedding-ready data. One canonical model and one
+shared JATS parser back every path.
+
+## When to use
+
+- The user gives a **preprint DOI or bioRxiv/medRxiv URL** and wants the full text or
+  a specific section (methods, results, …).
+- The user wants to **search** preprints by keyword, **title**, **abstract**, or
+  **author**, or **discover** them by topic/category/date.
+- The user wants a **corpus** of preprint full text chunked for embeddings/RAG.
+
+Do NOT use for published-journal full text in general (this is preprint-scoped), and
+do not scrape the bioRxiv website in bulk — use `ingest --source s3` for corpora.
+
+## Why full text (for grounded reasoning)
+
+Prefer full text over abstracts when the task needs real evidence: methods, quantities,
+protocols, datasets, and results live in the body. `get_fulltext` returns section-labeled
+text (`kind` = methods / results / discussion / …) with `source`, `version`, and `license`,
+so you can:
+- **cite precisely** — quote/attribute the exact section of a specific paper + version;
+- **verify claims** — check an assertion against the paper's Results, not its abstract;
+- **extract methods/parameters** — read the Methods section directly;
+- **stay within-license** — each Section/Chunk carries its license and a link-back source.
+This grounds reasoning in the primary source rather than a recalled (possibly stale) summary.
+
+## CLI
+
+```bash
+# One preprint -> structured sections (Europe PMC -> S3 router). Accepts DOI, doi.org
+# URL, or a bioRxiv/medRxiv content URL. --markdown for readable output; default JSON.
+preprint-fulltext get <DOI|URL> [--markdown] [--version N] [--source europepmc|s3|html] [--html]
+
+# Search -> one SearchHit per line (JSONL). --field scopes the query.
+preprint-fulltext search "<query>" [--field fulltext|title|abstract|author] [--source europepmc|openalex] [-n N]
+
+# Discover by topic/category/date -> SearchHit JSONL (OpenAlex by default).
+preprint-fulltext discover [--query Q] [--category C] [--since YYYY-MM] [--to YYYY-MM-DD] [-n N]
+
+# Bulk corpus -> Chunk JSONL/Parquet + <out>_manifest.jsonl (resumable, incremental).
+preprint-fulltext ingest <out.jsonl> --source s3|europepmc [--server biorxiv|medrxiv|both]
+    [--since YYYY-MM] [--redistribution] [--format jsonl|parquet] [-n N]
+```
+
+Notes for agents:
+- **arXiv:** `get` accepts an arXiv id (`2401.10515`), `arXiv:` prefix, an arxiv.org URL, or
+  the `10.48550/arXiv.*` DOI, and returns full text from arXiv's LaTeXML HTML (native →
+  ar5iv). `search --source arxiv` / `discover --source arxiv` query the arXiv API. arXiv is
+  **not** available in `ingest` (single-document only).
+- **Versions:** a DOI resolves to the **latest version by default** (openRxiv semantics).
+  Pass `--version N` for a specific one. Europe PMC only serves the latest indexed
+  version, so an explicit older version is fetched via the HTML source (`--source html
+  --version 1`) or S3. The retrieved version is reported in `preprint.version`.
+- `get` with no `--source` tries Europe PMC, then S3 (if AWS creds), and only tries the
+  public HTML page if you pass `--html`. Europe PMC full text exists for the **CC /
+  open-access subset**; all-rights-reserved preprints need S3 or `--html`.
+- `--field title`/`abstract` match **all terms** (not an exact phrase); `--field author`
+  matches an author name. For topical multi-word queries prefer `--field fulltext`.
+- `ingest --source s3` is **requester-pays** (needs AWS credentials and costs money;
+  ~$0.09/GB). `ingest --source europepmc` is free but only covers the CC subset.
+- `--redistribution` degrades non-redistributable works to metadata-only stubs.
+
+## Python
+
+```python
+from preprint_fulltext.pipeline.router import Router
+from preprint_fulltext.core.chunk import chunk_fulltext
+
+result = Router().get_fulltext("10.1101/2024.01.15.575000")  # -> GetResult
+ft = result.fulltext                     # FullText | None (result.reason if None)
+chunks = chunk_fulltext(ft)              # list[Chunk], embedding-ready
+```
+
+## MCP tools
+
+Run `preprint-fulltext-mcp` (stdio). Tools:
+- `search_preprints(query, source?, limit?)` → list of hits
+- `get_fulltext(doi, as_markdown?)` → structured sections (or Markdown)
+- `get_metadata(doi)` → bibliographic metadata (abstract only)
+- `resolve(doi_or_openalex_id)` → cross-identifier resolution
+
+Bulk `ingest` is intentionally not an MCP tool.
+
+## Data model (what you get back)
+
+- `FullText`: `preprint` + ordered `sections` (`kind` ∈ abstract|intro|methods|results|
+  discussion|other) + `retrieved_from`.
+- `Chunk` (the corpus record): `doi, version, chunk_id, section_kind, text,
+  token_count, char_start, char_end, license, source`. One JSONL line per chunk.
+- `SearchHit`: `title, doi?, openalex_id?, score?, source, snippet?, oa_url?, has_fulltext`.
+
+## Gotchas
+
+- openRxiv DOIs use prefix **`10.64898`** since 2025-12 (legacy **`10.1101`** before).
+- OpenAlex requires an API key (`OPENALEX_API_KEY`) since 2026-02-13; missing → HTTP 409.
+- Set `CONTACT_EMAIL` for the Europe PMC / OpenAlex polite pools.
+- Retrieved content keeps its own license; corpora are for your own TDM, not re-hosting.

preprint_fulltext-0.1.0/.github/copilot-instructions.md

@@ -0,0 +1,102 @@
+<!-- GENERATED from skills/preprint-fulltext/SKILL.md by scripts/build_agent_docs.py — do not edit. -->
+
+# preprint-fulltext
+
+A CLI + Python library + MCP server for retrieving bioRxiv/medRxiv preprint full text
+and turning it into structured, embedding-ready data. One canonical model and one
+shared JATS parser back every path.
+
+## When to use
+
+- The user gives a **preprint DOI or bioRxiv/medRxiv URL** and wants the full text or
+  a specific section (methods, results, …).
+- The user wants to **search** preprints by keyword, **title**, **abstract**, or
+  **author**, or **discover** them by topic/category/date.
+- The user wants a **corpus** of preprint full text chunked for embeddings/RAG.
+
+Do NOT use for published-journal full text in general (this is preprint-scoped), and
+do not scrape the bioRxiv website in bulk — use `ingest --source s3` for corpora.
+
+## Why full text (for grounded reasoning)
+
+Prefer full text over abstracts when the task needs real evidence: methods, quantities,
+protocols, datasets, and results live in the body. `get_fulltext` returns section-labeled
+text (`kind` = methods / results / discussion / …) with `source`, `version`, and `license`,
+so you can:
+- **cite precisely** — quote/attribute the exact section of a specific paper + version;
+- **verify claims** — check an assertion against the paper's Results, not its abstract;
+- **extract methods/parameters** — read the Methods section directly;
+- **stay within-license** — each Section/Chunk carries its license and a link-back source.
+This grounds reasoning in the primary source rather than a recalled (possibly stale) summary.
+
+## CLI
+
+```bash
+# One preprint -> structured sections (Europe PMC -> S3 router). Accepts DOI, doi.org
+# URL, or a bioRxiv/medRxiv content URL. --markdown for readable output; default JSON.
+preprint-fulltext get <DOI|URL> [--markdown] [--version N] [--source europepmc|s3|html] [--html]
+
+# Search -> one SearchHit per line (JSONL). --field scopes the query.
+preprint-fulltext search "<query>" [--field fulltext|title|abstract|author] [--source europepmc|openalex] [-n N]
+
+# Discover by topic/category/date -> SearchHit JSONL (OpenAlex by default).
+preprint-fulltext discover [--query Q] [--category C] [--since YYYY-MM] [--to YYYY-MM-DD] [-n N]
+
+# Bulk corpus -> Chunk JSONL/Parquet + <out>_manifest.jsonl (resumable, incremental).
+preprint-fulltext ingest <out.jsonl> --source s3|europepmc [--server biorxiv|medrxiv|both]
+    [--since YYYY-MM] [--redistribution] [--format jsonl|parquet] [-n N]
+```
+
+Notes for agents:
+- **arXiv:** `get` accepts an arXiv id (`2401.10515`), `arXiv:` prefix, an arxiv.org URL, or
+  the `10.48550/arXiv.*` DOI, and returns full text from arXiv's LaTeXML HTML (native →
+  ar5iv). `search --source arxiv` / `discover --source arxiv` query the arXiv API. arXiv is
+  **not** available in `ingest` (single-document only).
+- **Versions:** a DOI resolves to the **latest version by default** (openRxiv semantics).
+  Pass `--version N` for a specific one. Europe PMC only serves the latest indexed
+  version, so an explicit older version is fetched via the HTML source (`--source html
+  --version 1`) or S3. The retrieved version is reported in `preprint.version`.
+- `get` with no `--source` tries Europe PMC, then S3 (if AWS creds), and only tries the
+  public HTML page if you pass `--html`. Europe PMC full text exists for the **CC /
+  open-access subset**; all-rights-reserved preprints need S3 or `--html`.
+- `--field title`/`abstract` match **all terms** (not an exact phrase); `--field author`
+  matches an author name. For topical multi-word queries prefer `--field fulltext`.
+- `ingest --source s3` is **requester-pays** (needs AWS credentials and costs money;
+  ~$0.09/GB). `ingest --source europepmc` is free but only covers the CC subset.
+- `--redistribution` degrades non-redistributable works to metadata-only stubs.
+
+## Python
+
+```python
+from preprint_fulltext.pipeline.router import Router
+from preprint_fulltext.core.chunk import chunk_fulltext
+
+result = Router().get_fulltext("10.1101/2024.01.15.575000")  # -> GetResult
+ft = result.fulltext                     # FullText | None (result.reason if None)
+chunks = chunk_fulltext(ft)              # list[Chunk], embedding-ready
+```
+
+## MCP tools
+
+Run `preprint-fulltext-mcp` (stdio). Tools:
+- `search_preprints(query, source?, limit?)` → list of hits
+- `get_fulltext(doi, as_markdown?)` → structured sections (or Markdown)
+- `get_metadata(doi)` → bibliographic metadata (abstract only)
+- `resolve(doi_or_openalex_id)` → cross-identifier resolution
+
+Bulk `ingest` is intentionally not an MCP tool.
+
+## Data model (what you get back)
+
+- `FullText`: `preprint` + ordered `sections` (`kind` ∈ abstract|intro|methods|results|
+  discussion|other) + `retrieved_from`.
+- `Chunk` (the corpus record): `doi, version, chunk_id, section_kind, text,
+  token_count, char_start, char_end, license, source`. One JSONL line per chunk.
+- `SearchHit`: `title, doi?, openalex_id?, score?, source, snippet?, oa_url?, has_fulltext`.
+
+## Gotchas
+
+- openRxiv DOIs use prefix **`10.64898`** since 2025-12 (legacy **`10.1101`** before).
+- OpenAlex requires an API key (`OPENALEX_API_KEY`) since 2026-02-13; missing → HTTP 409.
+- Set `CONTACT_EMAIL` for the Europe PMC / OpenAlex polite pools.
+- Retrieved content keeps its own license; corpora are for your own TDM, not re-hosting.

preprint_fulltext-0.1.0/.github/workflows/live-smoke.yml

@@ -0,0 +1,28 @@
+name: live-smoke
+# Opt-in LIVE smoke against the real public APIs (Europe PMC, arXiv, bioRxiv/medRxiv JSON API).
+# Kept OUT of the normal `test` job so CI stays offline/free by default. Run it manually
+# before a release, or on the weekly schedule, to catch upstream API drift (e.g. an endpoint
+# change or a new DOI prefix). Uses the CONTACT_EMAIL repo variable for the polite pools.
+#
+# NOTE: requester-pays S3 live tests are NOT run here (cost); they stay behind
+# PREPRINT_FULLTEXT_LIVE_S3 and are exercised locally with credentials only.
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 6 * * 1"          # Mondays 06:00 UTC
+
+jobs:
+  live:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with: { python-version: "3.11" }
+      - name: Install
+        run: python -m pip install -e ".[dev]"
+      - name: Live smoke (Europe PMC + arXiv + bioRxiv/medRxiv metadata; OpenAlex if key set)
+        env:
+          PREPRINT_FULLTEXT_LIVE: "1"
+          CONTACT_EMAIL: ${{ vars.CONTACT_EMAIL }}
+          OPENALEX_API_KEY: ${{ secrets.OPENALEX_API_KEY }}   # optional; OpenAlex test skips if unset
+        run: pytest -m live -v

preprint_fulltext-0.1.0/.github/workflows/publish-pypi.yml

@@ -0,0 +1,26 @@
+name: publish-pypi
+# Publish preprint-fulltext to PyPI on a GitHub Release, via PyPI Trusted Publishing
+# (OIDC — no API token needed). ONE-TIME SETUP: on pypi.org add a Trusted Publisher for
+# project "preprint-fulltext" pointing at genecell/preprint-fulltext + this workflow, with
+# the environment field left blank. (A GitHub Environment is optional — add one on both
+# sides only if you want a manual-approval gate before publishing.)
+on:
+  release:
+    types: [published]
+  workflow_dispatch:
+
+jobs:
+  build-and-publish:
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write        # OIDC identity for Trusted Publishing
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with: { python-version: "3.11" }
+      - name: Build sdist + wheel
+        run: |
+          python -m pip install build
+          python -m build
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

preprint_fulltext-0.1.0/.github/workflows/publish-registry.yml

@@ -0,0 +1,41 @@
+name: publish-registry
+# Register preprint-fulltext in the official MCP Registry (registry.modelcontextprotocol.io)
+# via GitHub OIDC — no tokens. The io.github.genecell/* namespace is authorized by OIDC from
+# this repo (owner genecell). The registry verifies ownership by fetching the PyPI package's
+# `mcp-name:` marker, so run this ONLY AFTER the new version is live on PyPI.
+#
+# HOW TO RELEASE: cut a GitHub Release (publish-pypi.yml uploads to PyPI) → confirm the version
+# is live on pypi.org → then Actions tab → publish-registry → Run workflow.
+on:
+  workflow_dispatch:
+
+jobs:
+  publish-registry:
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write        # OIDC identity for the io.github.genecell namespace
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+      - name: Sync server.json version to pyproject.toml
+        run: |
+          python - <<'PY'
+          import re, json, pathlib
+          ver = re.search(r'^version\s*=\s*"([^"]+)"', pathlib.Path("pyproject.toml").read_text(), re.M).group(1)
+          p = pathlib.Path("server.json"); d = json.loads(p.read_text())
+          d["version"] = ver
+          for pkg in d.get("packages", []):
+              pkg["version"] = ver
+          p.write_text(json.dumps(d, indent=2) + "\n")
+          print("server.json version ->", ver)
+          PY
+      - name: Install mcp-publisher (latest linux/amd64)
+        run: |
+          url=$(curl -s https://api.github.com/repos/modelcontextprotocol/registry/releases/latest \
+            | grep -o 'https://[^"]*mcp-publisher[^"]*linux_amd64.tar.gz' | head -1)
+          echo "Downloading $url"
+          curl -L "$url" | tar xz mcp-publisher
+      - name: Publish to the MCP Registry
+        run: |
+          ./mcp-publisher login github-oidc
+          ./mcp-publisher publish

preprint_fulltext-0.1.0/.github/workflows/test.yml

@@ -0,0 +1,43 @@
+name: test
+# Offline test suite (respx mocks HTTP, moto mocks S3) + lint. No live network,
+# no AWS, no cost — matches the project's "green offline by default" guarantee.
+on:
+  push: { branches: [master, main] }
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+      - name: Install (dev extras)
+        run: python -m pip install -e ".[dev]"
+      - name: Lint
+        run: ruff check preprint_fulltext/
+      - name: Test (offline)
+        run: pytest -q
+      - name: Build sdist + wheel
+        run: |
+          python -m pip install build
+          python -m build
+
+  docs-sync:
+    # Fail if the generated agent docs are stale vs skills/preprint-fulltext/SKILL.md.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with: { python-version: "3.11" }
+      - name: Regenerate agent docs and check for drift
+        run: |
+          python scripts/build_agent_docs.py
+          git diff --exit-code -- AGENTS.md llms.txt .cursor .github/copilot-instructions.md \
+            || (echo "::error::agent docs are stale — run: python scripts/build_agent_docs.py" && exit 1)

preprint_fulltext-0.1.0/.gitignore

@@ -0,0 +1,33 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.venv/
+venv/
+
+# Tooling
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+htmlcov/
+
+# Project
+.cache/
+*.jsonl
+*.parquet
+_manifest.jsonl
+preprint-fulltext.toml
+
+# stato — version the captured modules, but not transient history
+.stato/.history/
+
+# OS
+.DS_Store
+
+# internal dev context (lives at the project root, not in the public package)
+.stato/
+CLAUDE.md

preprint_fulltext-0.1.0/AGENTS.md

@@ -0,0 +1,102 @@
+<!-- GENERATED from skills/preprint-fulltext/SKILL.md by scripts/build_agent_docs.py — do not edit. -->
+
+# preprint-fulltext
+
+A CLI + Python library + MCP server for retrieving bioRxiv/medRxiv preprint full text
+and turning it into structured, embedding-ready data. One canonical model and one
+shared JATS parser back every path.
+
+## When to use
+
+- The user gives a **preprint DOI or bioRxiv/medRxiv URL** and wants the full text or
+  a specific section (methods, results, …).
+- The user wants to **search** preprints by keyword, **title**, **abstract**, or
+  **author**, or **discover** them by topic/category/date.
+- The user wants a **corpus** of preprint full text chunked for embeddings/RAG.
+
+Do NOT use for published-journal full text in general (this is preprint-scoped), and
+do not scrape the bioRxiv website in bulk — use `ingest --source s3` for corpora.
+
+## Why full text (for grounded reasoning)
+
+Prefer full text over abstracts when the task needs real evidence: methods, quantities,
+protocols, datasets, and results live in the body. `get_fulltext` returns section-labeled
+text (`kind` = methods / results / discussion / …) with `source`, `version`, and `license`,
+so you can:
+- **cite precisely** — quote/attribute the exact section of a specific paper + version;
+- **verify claims** — check an assertion against the paper's Results, not its abstract;
+- **extract methods/parameters** — read the Methods section directly;
+- **stay within-license** — each Section/Chunk carries its license and a link-back source.
+This grounds reasoning in the primary source rather than a recalled (possibly stale) summary.
+
+## CLI
+
+```bash
+# One preprint -> structured sections (Europe PMC -> S3 router). Accepts DOI, doi.org
+# URL, or a bioRxiv/medRxiv content URL. --markdown for readable output; default JSON.
+preprint-fulltext get <DOI|URL> [--markdown] [--version N] [--source europepmc|s3|html] [--html]
+
+# Search -> one SearchHit per line (JSONL). --field scopes the query.
+preprint-fulltext search "<query>" [--field fulltext|title|abstract|author] [--source europepmc|openalex] [-n N]
+
+# Discover by topic/category/date -> SearchHit JSONL (OpenAlex by default).
+preprint-fulltext discover [--query Q] [--category C] [--since YYYY-MM] [--to YYYY-MM-DD] [-n N]
+
+# Bulk corpus -> Chunk JSONL/Parquet + <out>_manifest.jsonl (resumable, incremental).
+preprint-fulltext ingest <out.jsonl> --source s3|europepmc [--server biorxiv|medrxiv|both]
+    [--since YYYY-MM] [--redistribution] [--format jsonl|parquet] [-n N]
+```
+
+Notes for agents:
+- **arXiv:** `get` accepts an arXiv id (`2401.10515`), `arXiv:` prefix, an arxiv.org URL, or
+  the `10.48550/arXiv.*` DOI, and returns full text from arXiv's LaTeXML HTML (native →
+  ar5iv). `search --source arxiv` / `discover --source arxiv` query the arXiv API. arXiv is
+  **not** available in `ingest` (single-document only).
+- **Versions:** a DOI resolves to the **latest version by default** (openRxiv semantics).
+  Pass `--version N` for a specific one. Europe PMC only serves the latest indexed
+  version, so an explicit older version is fetched via the HTML source (`--source html
+  --version 1`) or S3. The retrieved version is reported in `preprint.version`.
+- `get` with no `--source` tries Europe PMC, then S3 (if AWS creds), and only tries the
+  public HTML page if you pass `--html`. Europe PMC full text exists for the **CC /
+  open-access subset**; all-rights-reserved preprints need S3 or `--html`.
+- `--field title`/`abstract` match **all terms** (not an exact phrase); `--field author`
+  matches an author name. For topical multi-word queries prefer `--field fulltext`.
+- `ingest --source s3` is **requester-pays** (needs AWS credentials and costs money;
+  ~$0.09/GB). `ingest --source europepmc` is free but only covers the CC subset.
+- `--redistribution` degrades non-redistributable works to metadata-only stubs.
+
+## Python
+
+```python
+from preprint_fulltext.pipeline.router import Router
+from preprint_fulltext.core.chunk import chunk_fulltext
+
+result = Router().get_fulltext("10.1101/2024.01.15.575000")  # -> GetResult
+ft = result.fulltext                     # FullText | None (result.reason if None)
+chunks = chunk_fulltext(ft)              # list[Chunk], embedding-ready
+```
+
+## MCP tools
+
+Run `preprint-fulltext-mcp` (stdio). Tools:
+- `search_preprints(query, source?, limit?)` → list of hits
+- `get_fulltext(doi, as_markdown?)` → structured sections (or Markdown)
+- `get_metadata(doi)` → bibliographic metadata (abstract only)
+- `resolve(doi_or_openalex_id)` → cross-identifier resolution
+
+Bulk `ingest` is intentionally not an MCP tool.
+
+## Data model (what you get back)
+
+- `FullText`: `preprint` + ordered `sections` (`kind` ∈ abstract|intro|methods|results|
+  discussion|other) + `retrieved_from`.
+- `Chunk` (the corpus record): `doi, version, chunk_id, section_kind, text,
+  token_count, char_start, char_end, license, source`. One JSONL line per chunk.
+- `SearchHit`: `title, doi?, openalex_id?, score?, source, snippet?, oa_url?, has_fulltext`.
+
+## Gotchas
+
+- openRxiv DOIs use prefix **`10.64898`** since 2025-12 (legacy **`10.1101`** before).
+- OpenAlex requires an API key (`OPENALEX_API_KEY`) since 2026-02-13; missing → HTTP 409.
+- Set `CONTACT_EMAIL` for the Europe PMC / OpenAlex polite pools.
+- Retrieved content keeps its own license; corpora are for your own TDM, not re-hosting.

preprint_fulltext-0.1.0/LICENSE

@@ -0,0 +1,40 @@
+BSD 3-Clause License
+
+Copyright (c) 2026, Min Dai and the preprint-fulltext maintainers
+(Gord Fishell Lab, Harvard Medical School / Broad Institute)
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+---
+
+NOTE ON RETRIEVED CONTENT: This BSD-3-Clause license covers the preprint-fulltext
+software only. It does NOT relicense any preprint full text the software
+retrieves. Preprint content remains under its author-selected license (e.g.
+CC-BY, CC-BY-NC, or all-rights-reserved). Corpora built with this tool are for
+the operator's own text/data mining under the openRxiv TDM terms; do not
+re-host or redistribute openRxiv full text, and link back to the openRxiv-hosted
+text in any tool built on it. See the openRxiv TDM terms for details.