litfetch 0.1.0__tar.gz

litfetch-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Centre for Population Genomics
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

litfetch-0.1.0/PKG-INFO

@@ -0,0 +1,230 @@
+Metadata-Version: 2.4
+Name: litfetch
+Version: 0.1.0
+Summary: Identifier -> the retrievable artifacts of a scholarly article: a pluggable source ladder + identifier resolvers.
+Author: Toby Sargeant
+License: MIT
+Project-URL: Repository, https://github.com/populationgenomics/litfetch
+Project-URL: Issues, https://github.com/populationgenomics/litfetch/issues
+Keywords: pubmed,pmc,full-text,jats,elsevier,supplementary
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: defusedxml>=0.7
+Requires-Dist: httpx>=0.28
+Provides-Extra: biorxiv
+Requires-Dist: curl_cffi>=0.7; extra == "biorxiv"
+Dynamic: license-file
+
+# litfetch
+
+Resolve a scholarly article identifier to its **retrievable artifacts** — the
+full-text body and any supplementary material — and fetch their bytes.
+
+litfetch is two cooperating seams:
+
+- a **fetch ladder** — pluggable `Fetcher` backends (PMC Open Access S3, Europe
+  PMC, Elsevier OA) tried in priority order; the first to serve the body wins,
+  returning a `Blob` (a `File` plus its bytes);
+- an optional **resolver layer** — pluggable `Resolver`s that enrich what you
+  know about a paper (`pmid` → `pmcid`/`doi`, etc.) so the ladder can act.
+
+You hand it an `ArticleIds` bundle (any of `pmid` / `pmcid` / `doi`). Resolution
+is **demand-driven**: a resolver only runs when the next fetcher needs an
+identifier you don't yet have, and runs at most once.
+
+An article is modelled as a **file-set**: a collection of `File` references (the
+body in its various media types, plus supplementary material, distinguished by
+`FileKind`), each hosted upstream. litfetch fetches the raw artifacts and reports
+their access terms; it does **not** render them. To turn a fetched JATS/Elsevier
+body into markdown, run [litdown](https://github.com/populationgenomics/litdown)
+on the bytes yourself (see [Render to markdown](#render-to-markdown)).
+
+The examples below are a tour; [`docs/api.md`](docs/api.md) is the full
+reference for the public surface.
+
+## Install
+
+```sh
+pip install litfetch
+```
+
+bioRxiv / medRxiv preprint full text needs a browser-fingerprint HTTP client,
+enabled by the `biorxiv` extra:
+
+```sh
+pip install 'litfetch[biorxiv]'
+```
+
+## Usage
+
+### Fetch the body
+
+Hand `fetch_body` an `ArticleIds`; the default ladder serves the first available
+body as a `Blob`:
+
+```python
+from litfetch import ArticleIds, fetch_body
+
+blob = await fetch_body(ArticleIds(pmcid='PMC5334499'))
+if blob:
+    print(blob.file.source, blob.file.media_type, len(blob.content))
+```
+
+### Render to markdown
+
+litfetch returns raw bytes, not markdown. Convert a JATS/Elsevier body with
+[litdown](https://github.com/populationgenomics/litdown) — you pick and pin the
+converter:
+
+```python
+import io
+import litdown
+from litfetch import ArticleIds, fetch_body
+
+blob = await fetch_body(ArticleIds(pmcid='PMC5334499'))
+if blob:
+    markdown = litdown.convert(io.BytesIO(blob.content))
+```
+
+### Inject your own resolver
+
+A resolver is an async `(ArticleIds, Http) -> ArticleIds` — the session running
+it supplies the `Http`. Enrich from whatever you have — a corpus client, a local
+cache, an API — and `merge` it in (this one ignores `Http`, hence `_http`):
+
+```python
+from litfetch import ArticleIds, Http, fetch_body
+
+async def my_resolver(ids: ArticleIds, _http: Http) -> ArticleIds:
+    if not ids.pmid:
+        return ids
+    pmcid, doi = await my_corpus.lookup(ids.pmid)
+    return ids.merge(ArticleIds(pmcid=pmcid, doi=doi))
+
+blob = await fetch_body(ArticleIds(pmid='29622564'), resolver=my_resolver)
+```
+
+### Use a bundled resolver
+
+Bundled resolvers are constructed with their config, then passed in the same
+slot. `chain(...)` composes several (yours first, fallbacks after); it stops
+once every identifier is known:
+
+```python
+from litfetch import ArticleIds, fetch_body
+from litfetch.resolvers import SemanticScholarResolver, NcbiIdConverterResolver, chain
+
+resolver = chain(
+    my_resolver,                              # your own
+    SemanticScholarResolver(api_key=S2_KEY),  # bundled
+    NcbiIdConverterResolver(tool='myapp'),    # bundled
+)
+blob = await fetch_body(ArticleIds(pmid='29622564'), resolver=resolver)
+```
+
+Polite-pool identification (NCBI/Crossref `email`, Unpaywall's required `email`)
+comes from a session `contact`, not a hardcoded default — set it on the session:
+`async with litfetch.Session(contact='you@example.org') as s: await s.fetch_body(...)`.
+
+`default_resolver()` is a batteries-included, keyless chain
+(Europe PMC search + NCBI ID Converter).
+
+### No resolver — you already hold the IDs
+
+A non-PubMed paper you only have a DOI for, plus your own Elsevier key:
+
+```python
+blob = await fetch_body(
+    ArticleIds(doi='10.1016/j.cell.2020.01.001'),
+    credentials={'elsevier_api_key': key},
+)
+```
+
+### Supplementary material
+
+`list_files` enumerates the file-set (references, no bytes); `fetch_file`
+materialises one:
+
+```python
+from litfetch import ArticleIds, FileKind, list_files, fetch_file
+
+files = await list_files(ArticleIds(pmcid='PMC5334499'), kind=FileKind.SUPPLEMENTARY)
+for file in files:
+    blob = await fetch_file(file)
+```
+
+### Access terms
+
+Read the licence from the fetched bytes, falling back to an access authority
+(Unpaywall) when the bytes carry none:
+
+```python
+from litfetch import extract_source_metadata, resolve_access
+
+meta = extract_source_metadata(blob)          # from the JATS/Elsevier bytes
+if meta.licence is None:
+    meta = await resolve_access(ArticleIds(doi='10.1016/j.cell.2020.01.001'))
+```
+
+### Resolvers stand alone
+
+Each resolver is usable on its own as a cross-reference tool, independent of
+fetching. A resolver is given the `Http` to use, so run it inside a session:
+
+```python
+from litfetch import ArticleIds, Session
+from litfetch.resolvers import SemanticScholarResolver
+
+async with Session() as s:
+    ids = await SemanticScholarResolver()(ArticleIds(doi='10.1016/j.cell.2020.01.001'), s)
+print(ids.pmid, ids.pmcid)
+```
+
+### Batch: one session, a scope per paper
+
+The one-shot functions above each open a throwaway session. For many papers,
+hold one `Session` (pooled connection, shared pacing) and open a `scope` per
+paper — the scope caches within itself, so a duplicate upstream call (e.g.
+Unpaywall for both licence and PDF) is fetched once:
+
+```python
+from litfetch import ArticleIds, Session
+
+async with Session() as session:
+    for pmid in pmids:
+        async with session.scope() as s:
+            blob = await s.fetch_body(ArticleIds(pmid=pmid))
+            access = await s.resolve_access(ArticleIds(pmid=pmid))
+```
+
+## Extending
+
+- **A new body fetcher:** implement the `Fetcher` protocol — a `name`, a
+  `requires: frozenset[str]` of the `ArticleIds` fields it needs, and an async
+  `fetch(ids, *, credentials, http)` returning a body `Blob` or `None`.
+  Add it to a `fetchers=` list (or your own `default_fetchers`).
+- **A new file source:** implement the `FileSource` protocol — a `name`, and
+  async `list_files(ids, ...)` / `fetch_file(file, ...)` — to enumerate and
+  materialise an article's file-set (body renditions and supplementary alike).
+- **A new resolver:** write an async `ArticleIds -> ArticleIds` that fills gaps
+  via `ArticleIds.merge` and never overwrites a known id.
+
+## Development
+
+```sh
+uv sync
+uv run ruff check . && uv run ruff format --check .
+uv run pyright
+uv run pytest
+```

litfetch-0.1.0/README.md

@@ -0,0 +1,202 @@
+# litfetch
+
+Resolve a scholarly article identifier to its **retrievable artifacts** — the
+full-text body and any supplementary material — and fetch their bytes.
+
+litfetch is two cooperating seams:
+
+- a **fetch ladder** — pluggable `Fetcher` backends (PMC Open Access S3, Europe
+  PMC, Elsevier OA) tried in priority order; the first to serve the body wins,
+  returning a `Blob` (a `File` plus its bytes);
+- an optional **resolver layer** — pluggable `Resolver`s that enrich what you
+  know about a paper (`pmid` → `pmcid`/`doi`, etc.) so the ladder can act.
+
+You hand it an `ArticleIds` bundle (any of `pmid` / `pmcid` / `doi`). Resolution
+is **demand-driven**: a resolver only runs when the next fetcher needs an
+identifier you don't yet have, and runs at most once.
+
+An article is modelled as a **file-set**: a collection of `File` references (the
+body in its various media types, plus supplementary material, distinguished by
+`FileKind`), each hosted upstream. litfetch fetches the raw artifacts and reports
+their access terms; it does **not** render them. To turn a fetched JATS/Elsevier
+body into markdown, run [litdown](https://github.com/populationgenomics/litdown)
+on the bytes yourself (see [Render to markdown](#render-to-markdown)).
+
+The examples below are a tour; [`docs/api.md`](docs/api.md) is the full
+reference for the public surface.
+
+## Install
+
+```sh
+pip install litfetch
+```
+
+bioRxiv / medRxiv preprint full text needs a browser-fingerprint HTTP client,
+enabled by the `biorxiv` extra:
+
+```sh
+pip install 'litfetch[biorxiv]'
+```
+
+## Usage
+
+### Fetch the body
+
+Hand `fetch_body` an `ArticleIds`; the default ladder serves the first available
+body as a `Blob`:
+
+```python
+from litfetch import ArticleIds, fetch_body
+
+blob = await fetch_body(ArticleIds(pmcid='PMC5334499'))
+if blob:
+    print(blob.file.source, blob.file.media_type, len(blob.content))
+```
+
+### Render to markdown
+
+litfetch returns raw bytes, not markdown. Convert a JATS/Elsevier body with
+[litdown](https://github.com/populationgenomics/litdown) — you pick and pin the
+converter:
+
+```python
+import io
+import litdown
+from litfetch import ArticleIds, fetch_body
+
+blob = await fetch_body(ArticleIds(pmcid='PMC5334499'))
+if blob:
+    markdown = litdown.convert(io.BytesIO(blob.content))
+```
+
+### Inject your own resolver
+
+A resolver is an async `(ArticleIds, Http) -> ArticleIds` — the session running
+it supplies the `Http`. Enrich from whatever you have — a corpus client, a local
+cache, an API — and `merge` it in (this one ignores `Http`, hence `_http`):
+
+```python
+from litfetch import ArticleIds, Http, fetch_body
+
+async def my_resolver(ids: ArticleIds, _http: Http) -> ArticleIds:
+    if not ids.pmid:
+        return ids
+    pmcid, doi = await my_corpus.lookup(ids.pmid)
+    return ids.merge(ArticleIds(pmcid=pmcid, doi=doi))
+
+blob = await fetch_body(ArticleIds(pmid='29622564'), resolver=my_resolver)
+```
+
+### Use a bundled resolver
+
+Bundled resolvers are constructed with their config, then passed in the same
+slot. `chain(...)` composes several (yours first, fallbacks after); it stops
+once every identifier is known:
+
+```python
+from litfetch import ArticleIds, fetch_body
+from litfetch.resolvers import SemanticScholarResolver, NcbiIdConverterResolver, chain
+
+resolver = chain(
+    my_resolver,                              # your own
+    SemanticScholarResolver(api_key=S2_KEY),  # bundled
+    NcbiIdConverterResolver(tool='myapp'),    # bundled
+)
+blob = await fetch_body(ArticleIds(pmid='29622564'), resolver=resolver)
+```
+
+Polite-pool identification (NCBI/Crossref `email`, Unpaywall's required `email`)
+comes from a session `contact`, not a hardcoded default — set it on the session:
+`async with litfetch.Session(contact='you@example.org') as s: await s.fetch_body(...)`.
+
+`default_resolver()` is a batteries-included, keyless chain
+(Europe PMC search + NCBI ID Converter).
+
+### No resolver — you already hold the IDs
+
+A non-PubMed paper you only have a DOI for, plus your own Elsevier key:
+
+```python
+blob = await fetch_body(
+    ArticleIds(doi='10.1016/j.cell.2020.01.001'),
+    credentials={'elsevier_api_key': key},
+)
+```
+
+### Supplementary material
+
+`list_files` enumerates the file-set (references, no bytes); `fetch_file`
+materialises one:
+
+```python
+from litfetch import ArticleIds, FileKind, list_files, fetch_file
+
+files = await list_files(ArticleIds(pmcid='PMC5334499'), kind=FileKind.SUPPLEMENTARY)
+for file in files:
+    blob = await fetch_file(file)
+```
+
+### Access terms
+
+Read the licence from the fetched bytes, falling back to an access authority
+(Unpaywall) when the bytes carry none:
+
+```python
+from litfetch import extract_source_metadata, resolve_access
+
+meta = extract_source_metadata(blob)          # from the JATS/Elsevier bytes
+if meta.licence is None:
+    meta = await resolve_access(ArticleIds(doi='10.1016/j.cell.2020.01.001'))
+```
+
+### Resolvers stand alone
+
+Each resolver is usable on its own as a cross-reference tool, independent of
+fetching. A resolver is given the `Http` to use, so run it inside a session:
+
+```python
+from litfetch import ArticleIds, Session
+from litfetch.resolvers import SemanticScholarResolver
+
+async with Session() as s:
+    ids = await SemanticScholarResolver()(ArticleIds(doi='10.1016/j.cell.2020.01.001'), s)
+print(ids.pmid, ids.pmcid)
+```
+
+### Batch: one session, a scope per paper
+
+The one-shot functions above each open a throwaway session. For many papers,
+hold one `Session` (pooled connection, shared pacing) and open a `scope` per
+paper — the scope caches within itself, so a duplicate upstream call (e.g.
+Unpaywall for both licence and PDF) is fetched once:
+
+```python
+from litfetch import ArticleIds, Session
+
+async with Session() as session:
+    for pmid in pmids:
+        async with session.scope() as s:
+            blob = await s.fetch_body(ArticleIds(pmid=pmid))
+            access = await s.resolve_access(ArticleIds(pmid=pmid))
+```
+
+## Extending
+
+- **A new body fetcher:** implement the `Fetcher` protocol — a `name`, a
+  `requires: frozenset[str]` of the `ArticleIds` fields it needs, and an async
+  `fetch(ids, *, credentials, http)` returning a body `Blob` or `None`.
+  Add it to a `fetchers=` list (or your own `default_fetchers`).
+- **A new file source:** implement the `FileSource` protocol — a `name`, and
+  async `list_files(ids, ...)` / `fetch_file(file, ...)` — to enumerate and
+  materialise an article's file-set (body renditions and supplementary alike).
+- **A new resolver:** write an async `ArticleIds -> ArticleIds` that fills gaps
+  via `ArticleIds.merge` and never overwrites a known id.
+
+## Development
+
+```sh
+uv sync
+uv run ruff check . && uv run ruff format --check .
+uv run pyright
+uv run pytest
+```

litfetch-0.1.0/litfetch/__init__.py

@@ -0,0 +1,92 @@
+"""litfetch: identifier -> the retrievable artifacts of a scholarly article.
+
+Hand :func:`fetch_body` an :class:`ArticleIds` bundle (any of pmid / pmcid / doi)
+and, optionally, a :data:`~litfetch.resolvers.Resolver` to fill in missing
+identifiers on demand.  A :class:`~litfetch.fetchers.Fetcher` ladder is tried in
+priority order; the first to serve the body yields a :class:`Blob` (a
+:class:`File` plus its bytes).  Supplementary material is discovered with
+:func:`list_files` and fetched with :func:`fetch_file`.
+
+An article is modelled as a *file-set*: a collection of :class:`File` references
+(body renditions and supplementary material, by :class:`FileKind`) sharing one
+identity, each hosted upstream.  litfetch fetches the raw artifacts and reports
+their access terms (:class:`SourceMetadata`); rendering them (e.g. XML ->
+markdown via litdown) and storing them are the consumer's concern.  The bundled
+identifier resolvers (Europe PMC, NCBI ID Converter, Semantic Scholar) live in
+:mod:`litfetch.resolvers`; file-set listing and fetching live in
+:mod:`litfetch.fetchers`.
+"""
+
+from __future__ import annotations
+
+from litfetch._http import Http, Rate, RetryPolicy
+from litfetch.artifacts import (
+    INSTITUTIONAL,
+    Blob,
+    File,
+    FileKind,
+    SourceMetadata,
+)
+from litfetch.fetchers import (
+    BiorxivFetcher,
+    CrossrefFileSource,
+    ElsevierFetcher,
+    EuropePmcFetcher,
+    Fetcher,
+    FileSource,
+    PmcOaFetcher,
+    SemanticScholarFileSource,
+    SpringerFetcher,
+    SpringerFileSource,
+    UnpaywallFileSource,
+    default_fetchers,
+    default_file_sources,
+)
+from litfetch.ids import ArticleIds
+from litfetch.relations import Related, RelationType
+from litfetch.sessions import (
+    Session,
+    fetch_body,
+    fetch_file,
+    list_files,
+    related_ids,
+    resolve_access,
+)
+from litfetch.source_metadata import extract_source_metadata
+
+__version__ = '0.1.0'
+
+__all__ = [
+    'INSTITUTIONAL',
+    'ArticleIds',
+    'BiorxivFetcher',
+    'Blob',
+    'CrossrefFileSource',
+    'ElsevierFetcher',
+    'EuropePmcFetcher',
+    'Fetcher',
+    'File',
+    'FileKind',
+    'FileSource',
+    'Http',
+    'PmcOaFetcher',
+    'Rate',
+    'Related',
+    'RelationType',
+    'RetryPolicy',
+    'SemanticScholarFileSource',
+    'Session',
+    'SourceMetadata',
+    'SpringerFetcher',
+    'SpringerFileSource',
+    'UnpaywallFileSource',
+    '__version__',
+    'default_fetchers',
+    'default_file_sources',
+    'extract_source_metadata',
+    'fetch_body',
+    'fetch_file',
+    'list_files',
+    'related_ids',
+    'resolve_access',
+]

litfetch-0.1.0/litfetch/_doi.py

@@ -0,0 +1,78 @@
+"""DOI validation and URL-safe path encoding.
+
+Several sources interpolate a DOI into an upstream URL path (Unpaywall,
+Crossref, Semantic Scholar, bioRxiv, the future doi.org resolve).  Doing so
+raw is wrong twice over: a DOI suffix may contain ``?``, ``#``, spaces, or
+``/`` -- which truncate or reshape the URL -- and a crafted ``.``/``..``
+segment is a path-traversal vector.  :func:`encode_doi_path` is the one safe
+way to place a DOI in a URL path; :func:`normalize_and_validate_doi` is the
+shape gate it builds on.
+"""
+
+from __future__ import annotations
+
+import re
+import urllib.parse
+
+# A DOI is ``10.<registrant>/<suffix>``: the registrant is one or more digits
+# with optional dot-separated sub-elements (e.g. ``10.1000.10``); the suffix is
+# any non-empty string.  The digit count is left open -- the common 4-9 range is
+# Crossref's observed corpus, not a spec rule -- so an unusual registrant is not
+# rejected.  DOIs are case-insensitive, so the prefix match is too; the value is
+# returned unchanged (suffixes are case-sensitive for many registrants).
+_DOI_RE = re.compile(r'^10\.\d+(?:\.\d+)*/.+$', re.IGNORECASE)
+
+# Decorations a caller-supplied DOI may arrive with; stripped before validation.
+_RESOLVER_PREFIXES = ('https://doi.org/', 'http://doi.org/', 'https://dx.doi.org/', 'http://dx.doi.org/')
+
+
+def normalize_and_validate_doi(doi: str) -> str:
+    """Return the bare, validated DOI, stripping common decorations.
+
+    Accepts a DOI carrying surrounding whitespace, a ``doi:`` scheme, or a
+    resolver URL prefix (``https://doi.org/``, ``http://dx.doi.org/``) and
+    returns the bare ``10.xxxx/suffix`` form.
+
+    Args:
+        doi: The DOI to normalise, possibly decorated.
+
+    Returns:
+        The bare DOI.
+
+    Raises:
+        ValueError: If the result is not a syntactically valid DOI.
+    """
+    candidate = doi.strip()
+    lowered = candidate.lower()
+    for prefix in _RESOLVER_PREFIXES:
+        if lowered.startswith(prefix):
+            candidate = candidate[len(prefix) :]
+            break
+    if candidate.lower().startswith('doi:'):
+        candidate = candidate[len('doi:') :].strip()
+    if not _DOI_RE.match(candidate):
+        raise ValueError(f'not a valid DOI: {doi!r}')
+    return candidate
+
+
+def encode_doi_path(doi: str) -> str:
+    """Percent-encode a validated DOI for safe interpolation into a URL path.
+
+    Validates via :func:`normalize_and_validate_doi`, then percent-encodes each
+    ``/``-separated segment -- so a suffix ``/``, ``?``, ``#``, or space cannot
+    reshape the URL -- and rejects a ``.`` or ``..`` segment (path traversal).
+
+    Args:
+        doi: The DOI to encode, possibly decorated.
+
+    Returns:
+        The encoded DOI, ready to interpolate after a URL's path separator.
+
+    Raises:
+        ValueError: If the DOI is invalid or contains a dot-segment.
+    """
+    normalized = normalize_and_validate_doi(doi)
+    segments = normalized.split('/')
+    if any(segment in ('.', '..') for segment in segments):
+        raise ValueError(f'DOI contains a path-traversal segment: {doi!r}')
+    return '/'.join(urllib.parse.quote(segment, safe='') for segment in segments)