linkml-reference-validator 0.2.1rc1__tar.gz → 0.2.1rc3__tar.gz

{linkml_reference_validator-0.2.1rc1 → linkml_reference_validator-0.2.1rc3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: linkml_reference_validator
-Version: 0.2.1rc1
+Version: 0.2.1rc3
 Summary: Validation of supporting text from references and publications
 Author-email: Chris Mungall <cjmungall@lbl.gov>
 License-Expression: Apache-2.0
@@ -395,10 +395,36 @@ reference:
 - `PMID:12345678`
 - `12345678` (assumes PMID)
 
-### Coming Soon
+#### Preprints
 
-- **DOI** - `DOI:10.1038/nature12345`
-- **URLs** - Web pages and online documents
+Preprints are supported as first-class references:
+
+```yaml
+reference:
+  id: DOI:10.1101/2024.01.01.573333   # by DOI
+  supporting_text: "the kinase phosphorylates the substrate in vitro"
+```
+
+```yaml
+reference:
+  id: PPR:PPR123456                    # by Europe PMC preprint id
+  supporting_text: "the kinase phosphorylates the substrate in vitro"
+```
+
+**Fetches:**
+- Abstract and metadata (title, authors, server, year, DOI)
+- Full-text body via the Europe PMC `fulltextRepo` PDF route (`epmc_preprint`
+  provider), for the Europe-PMC-ingested subset
+- `is_preprint` / `peer_review_status` flags, so downstream KBs can apply
+  "not sole support" policies
+
+**ID Formats:**
+- `DOI:10.1101/...` (Crossref `posted-content` is detected as a preprint)
+- `PPR:PPR123456` (Europe PMC `SRC:PPR` id)
+- NIH Preprint Pilot preprints carry real PMIDs and resolve via the `PMID:` path
+
+See the [full-text guide](docs/how-to/fetch-full-text-and-pdfs.md#preprints) for
+details and coverage caveats.
 
 ---
 

{linkml_reference_validator-0.2.1rc1 → linkml_reference_validator-0.2.1rc3}/README.md

@@ -374,10 +374,36 @@ reference:
 - `PMID:12345678`
 - `12345678` (assumes PMID)
 
-### Coming Soon
+#### Preprints
 
-- **DOI** - `DOI:10.1038/nature12345`
-- **URLs** - Web pages and online documents
+Preprints are supported as first-class references:
+
+```yaml
+reference:
+  id: DOI:10.1101/2024.01.01.573333   # by DOI
+  supporting_text: "the kinase phosphorylates the substrate in vitro"
+```
+
+```yaml
+reference:
+  id: PPR:PPR123456                    # by Europe PMC preprint id
+  supporting_text: "the kinase phosphorylates the substrate in vitro"
+```
+
+**Fetches:**
+- Abstract and metadata (title, authors, server, year, DOI)
+- Full-text body via the Europe PMC `fulltextRepo` PDF route (`epmc_preprint`
+  provider), for the Europe-PMC-ingested subset
+- `is_preprint` / `peer_review_status` flags, so downstream KBs can apply
+  "not sole support" policies
+
+**ID Formats:**
+- `DOI:10.1101/...` (Crossref `posted-content` is detected as a preprint)
+- `PPR:PPR123456` (Europe PMC `SRC:PPR` id)
+- NIH Preprint Pilot preprints carry real PMIDs and resolve via the `PMID:` path
+
+See the [full-text guide](docs/how-to/fetch-full-text-and-pdfs.md#preprints) for
+details and coverage caveats.
 
 ---
 

linkml_reference_validator-0.2.1rc3/docs/how-to/export-cache-to-zotero.md

@@ -0,0 +1,114 @@
+# Export a Reference Cache to Zotero
+
+This workflow turns a checked-in public reference cache into a Zotero collection,
+asks Zotero to find available PDFs, and then inventories the resulting attachments
+without putting closed manuscripts into the project repository.
+
+The three stores remain separate:
+
+| Store | Purpose | Safe to commit? |
+|---|---|---:|
+| Project `references_cache` | Reproducible public validation evidence | Yes |
+| Zotero library | User-managed references and PDF attachments | No |
+| Private research cache | Extracted user-library text for agent context | No |
+
+Normal validation reads only the first store.
+
+## 1. Export metadata from the public cache
+
+Run the exporter from the `linkml-reference-validator` checkout. For dismech:
+
+```bash
+uv run linkml-reference-validator cache export \
+  --cache-dir /Users/cjm/repos/dismech/references_cache \
+  --needs-full-text \
+  --output /Users/cjm/Downloads/dismech-zotero.json
+```
+
+`--needs-full-text` is the default. It excludes cache entries that already have
+full text, excludes non-publication identifiers, and deduplicates by normalized
+DOI and then PMID. The CSL JSON output uses an explicit metadata allowlist:
+
+- title and authors;
+- journal and year;
+- DOI; or
+- PMID and its public PubMed URL.
+
+It never contains cached article text, excerpts, provenance, PDFs, local paths,
+or private-cache data. The command refuses to replace an existing output file;
+use `--force` only after checking the destination. Use `--all` if you explicitly
+want eligible references that already contain full text.
+
+## 2. Import the file into Zotero
+
+In Zotero:
+
+1. Choose **File → Import**.
+2. Choose **A file**.
+3. Select `dismech-zotero.json`.
+4. Keep the imported records in a project-specific collection such as
+   **dismech**.
+
+CSL JSON is a standard format supported by Zotero. Importing the file adds only
+bibliographic records; it does not copy anything from the validator's caches.
+
+## 3. Ask Zotero to find PDFs
+
+Select one or more top-level journal articles in the center item list,
+right-click the selection, and choose **Find Full Text**. It is a built-in Zotero
+command, not a plugin.
+
+![Zotero Find Full Text processing a batch of imported references](../assets/zotero-find-full-text.png)
+
+The command appears for an eligible parent item that has a DOI and does not
+already have a file attachment. It may also be unavailable in a group library
+where files cannot be added. If it is absent, first test a single article in
+your personal library and confirm that its DOI field is populated.
+
+For a large import, start with a modest batch rather than all records at once.
+Publishers can temporarily rate-limit repeated requests, and a later retry may
+find additional files. Zotero reports **Full Text PDF** or **No file found** for
+each attempted item.
+
+## 4. Inventory the enriched Zotero library
+
+Once Zotero finishes, scan the project cache again:
+
+```bash
+uv run linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir /Users/cjm/repos/dismech/references_cache \
+  --dry-run
+```
+
+This read-only scan matches exact DOI, PMID, or PMCID identifiers and reports
+which cached references now have usable Zotero text or PDF attachments. It does
+not invoke **Find Full Text** and does not modify Zotero.
+
+After reviewing the matches, materialize them into the separate private research
+cache:
+
+```bash
+uv run linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir /Users/cjm/repos/dismech/references_cache \
+  --apply
+```
+
+Apply mode leaves `references_cache` unchanged. Private content goes to
+`~/.cache/linkml-reference-validator/private` by default with owner-only
+permissions. Agents may inspect it for context, but validation cannot use it as
+evidence.
+
+## 5. Validate reproducibly
+
+Run validation normally. Results remain reproducible because the validator
+reads the public project cache and accepts only public full-text provider results.
+Neither the contents of Zotero nor the machine-specific private cache affect
+validation.
+
+## See also
+
+- [Fetching Full Text and PDFs](fetch-full-text-and-pdfs.md)
+- [Zotero: importing standardized formats](https://www.zotero.org/support/kb/importing_standardized_formats)
+- [Zotero: adding files to a library](https://www.zotero.org/support/attaching_files)

linkml_reference_validator-0.2.1rc3/docs/how-to/fetch-full-text-and-pdfs.md

@@ -0,0 +1,287 @@
+# Fetching Full Text and PDFs
+
+This guide explains how the validator obtains the **full text** of a reference
+(not just its abstract or metadata) by trying a chain of full-text providers,
+optionally downloading and extracting text from PDFs.
+
+## Overview
+
+When a metadata source (Crossref, PubMed, DataCite, etc.) returns only an
+abstract or title, the validator can fall through to a **full-text provider
+chain**. Each provider attempts to *locate* an open-access copy of the
+reference. The first provider that yields usable full text wins; the located
+resource is downloaded and, if it is a PDF, the text is extracted and cached
+alongside the reference.
+
+This behaviour is **on by default**.
+
+## The provider chain
+
+The validator tries providers in order until one returns usable full text.
+The default order is:
+
+```
+pmc → epmc_preprint → unpaywall → openalex
+```
+
+- **`pmc`** — PubMed Central open-access subset (XML/HTML full text for many
+  biomedical articles).
+- **`epmc_preprint`** — Europe PMC preprint body text via the `fulltextRepo`
+  PDF route (see [Preprints](#preprints) below).
+- **`unpaywall`** — Unpaywall open-access lookup by DOI (often a publisher or
+  repository PDF).
+- **`openalex`** — OpenAlex open-access location (PDF/HTML).
+
+Providers are skipped silently if they have nothing for a given reference, so
+the chain degrades gracefully: a miss in `pmc` simply moves on to `unpaywall`,
+and so on. If no provider yields full text, the reference keeps whatever
+metadata-only content it already had.
+
+## Configuration
+
+Full-text fetching is controlled by the following configuration keys (set in
+your config YAML or on the `ReferenceValidationConfig` object):
+
+| Key | Default | Description |
+|-----|---------|-------------|
+| `fetch_full_text` | `true` | Attempt to obtain full text via the provider chain when a metadata source does not already return full text. |
+| `full_text_providers` | `[pmc, epmc_preprint, unpaywall, openalex]` | Ordered list of provider names to try until one yields usable full text. |
+| `private_cache_dir` | `~/.cache/linkml-reference-validator/private` | Separate research cache for closed/user-library full text; never read by validation. |
+| `pdf_backend` | `pypdf` | Name of the PDF text-extraction backend. |
+| `download_pdfs` | `true` | If true, persist downloaded PDFs to the files cache directory. |
+| `full_text_providers_file` | `null` | Optional path to a YAML file defining custom full-text providers. |
+| `zotero_base_url` | `http://localhost:23119/api/users/0` | Read-only Zotero local API library used by the opt-in `zotero` provider. |
+
+Two existing keys are also reused by the full-text machinery:
+
+| Key | Description |
+|-----|-------------|
+| `email` | Used for "polite pool" access to providers such as Unpaywall and OpenAlex (and for Crossref/Entrez). Set this for more reliable access. |
+| `max_supplementary_file_size` | Upper bound (in bytes) on individual file downloads; a downloaded full-text PDF larger than this limit is not persisted. |
+
+Example config YAML:
+
+```yaml
+email: you@example.org
+fetch_full_text: true
+full_text_providers:
+  - pmc
+  - epmc_preprint
+  - unpaywall
+  - openalex
+pdf_backend: pypdf
+download_pdfs: true
+```
+
+## Private manuscripts in Zotero
+
+The opt-in **`zotero`** provider can find non-open manuscripts that are already
+in your personal Zotero library. It matches exact DOI, PMID, or PMCID identifiers;
+it does not automatically accept fuzzy title matches.
+
+Zotero must be running, and its local API must be enabled in Zotero's advanced
+settings. Inventory the reference cache before changing it:
+
+```bash
+linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir references_cache \
+  --dry-run
+```
+
+The report distinguishes `found`, `not_found`, `already_full_text`, and `error`.
+Dry-run is the default and never changes cache files. After reviewing exact
+matches, apply usable text with:
+
+```bash
+linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir references_cache \
+  --apply
+```
+
+Apply mode does **not** modify `references_cache`. It writes the enriched entry
+to a separate research cache at `~/.cache/linkml-reference-validator/private`
+by default. Agents may inspect that cache for background context, but ordinary
+validation never reads it and never cites it as evidence. Validation therefore
+has the same inputs locally, in CI, and on another contributor's machine.
+
+To keep the research cache in a private repository or another secured location:
+
+```bash
+linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir references_cache \
+  --private-cache-dir /path/to/private-reference-cache \
+  --apply
+```
+
+The same location can be configured for all commands:
+
+```yaml
+cache_dir: references_cache
+private_cache_dir: /path/to/private-reference-cache
+```
+
+Zotero-indexed text is preferred. If Zotero has not indexed an attachment, the
+provider downloads it from Zotero's local PDF endpoint and sends it through the
+normal PDF extractor. The source attachment key and `user_library` access type
+are persisted, but the ephemeral localhost URL is not.
+
+!!! warning "Keep research caches private"
+
+    Private cache entries contain extracted manuscript text and may contain a
+    copied PDF. Do not place this cache in a public repository or share it
+    unless you have permission to redistribute that content.
+
+Do not add `zotero` to `full_text_providers`. Even if it is configured there,
+ordinary validation rejects its `user_library` result and continues to public
+providers. Use `cache enrich` when you want to search Zotero.
+
+Support for reproducible validation backed by private manuscripts is a future,
+separate feature. The proposed design is a trusted offline job that emits a
+minimal signed excerpt attestation into the public cache. The validator would
+verify the signature using a checked-in public key; neither agents nor CI would
+have the signing key or permission to generate those files directly.
+
+## Export a cache to a Zotero collection
+
+Export references that still need full text as metadata-only CSL JSON:
+
+```bash
+linkml-reference-validator cache export \
+  --cache-dir references_cache \
+  --format csl-json \
+  --needs-full-text \
+  --output project-zotero.json
+```
+
+The default `--needs-full-text` mode excludes cache entries that already contain
+full text. It also excludes non-publication identifiers, deduplicates by
+normalized DOI and then PMID, and exports only an explicit metadata allowlist:
+title, authors, journal, year, DOI, PMID, and PubMed URL. Cached article text,
+excerpts, provenance, PDFs, and local paths are never exported. Use `--all` only
+when you deliberately want eligible records that already have full text.
+
+The exporter refuses to replace an existing output file unless `--force` is
+given. Review the reported counts, then in Zotero choose **File → Import → A
+file**, select the JSON file, and place the imported items in a project-specific
+collection. Zotero supports CSL JSON as a standard import format. After import,
+select the collection and run **Find Full Text**, then rerun `cache enrich`.
+
+See [Zotero's standardized-format import instructions](https://www.zotero.org/support/kb/importing_standardized_formats).
+For the complete workflow with screenshots, see
+[Export a Reference Cache to Zotero](export-cache-to-zotero.md).
+
+## Preprints
+
+Preprints are first-class references. They are increasingly cited for early
+mechanistic findings, but they sit in two distinct universes that need
+different handling:
+
+- **NIH Preprint Pilot** — a subset of preprints carry real PMIDs, so they
+  resolve through the normal `PMID:` path with no special handling.
+- **Europe PMC `SRC:PPR`** — the much larger preprint firehose is DOI-only
+  (no PMIDs). These resolve by DOI through Crossref, or directly by their
+  Europe PMC preprint id using the `PPR:` prefix (e.g. `PPR:PPR123456`).
+
+### Full-text retrieval
+
+Preprint bodies are **not** served by the PMC `fullTextXML` endpoint — even the
+~7% with a PMCID expose only abstract/title/supplementary in PMC, and direct
+`biorxiv.org/...full.pdf` links sit behind a bot challenge. The route that
+works for automated clients is the Europe PMC `fulltextRepo` PDF, used by the
+`epmc_preprint` provider: it resolves a preprint by DOI (or `SRC:PPR` id),
+finds the open `fulltextRepo` PDF URL, and downloads it. The PDF's `%PDF-`
+magic bytes are validated before extraction, so a stale-filename error blob is
+rejected rather than fed to the PDF parser, and the chain falls through
+gracefully.
+
+Coverage note: the `epmc_preprint` route covers the Europe-PMC-ingested
+preprint subset. The newest openRxiv DOIs (prefix `10.64898`) are not yet in
+Europe PMC, so very recent preprints may resolve to metadata only until they
+are ingested.
+
+### Preprint / peer-review status
+
+When a reference is detected as a preprint — via the Crossref `posted-content`
+type or a Europe PMC `SRC:PPR` record — the cached reference records it:
+
+| Field | Description |
+|-------|-------------|
+| `is_preprint` | `true` for a preprint, `false` when the source authoritatively classifies it as not a preprint (e.g. a Crossref `journal-article`), and unset when the type is unknown. |
+| `peer_review_status` | `preprint` for preprints; left unset otherwise. |
+
+Downstream knowledge bases can use these fields to apply policies such as "a
+preprint may not be the sole support for a claim". A non-preprint is not
+positively asserted as `peer_reviewed`; `peer_review_status` stays unset.
+
+## CLI flag
+
+Every `validate` subcommand accepts a `--full-text/--no-full-text` flag that
+toggles `fetch_full_text` for that run (default: on):
+
+```bash
+# Default: full-text chain is tried
+linkml-reference-validator validate data data.yaml \
+  --schema schema.yaml --target-class Statement
+
+# Disable full-text fetching (metadata only)
+linkml-reference-validator validate data data.yaml \
+  --schema schema.yaml --target-class Statement --no-full-text
+```
+
+## Custom providers
+
+You can declare your own JSON-API-backed provider in a YAML file and point
+`full_text_providers_file` at it. Each entry under `full_text_providers` is
+keyed by the provider name and must supply a `url_template`. The template may
+reference `{doi}`, `{pmid}`, or `{pmcid}`. `location_field` is a JSONPath into
+the response that holds the URL of the full-text resource; `format_hint` tells
+the validator how to treat it; and `headers` are sent with the request, with
+`${ENV_VAR}` placeholders interpolated from the environment.
+
+```yaml
+full_text_providers:
+  myrepo:
+    url_template: https://api.example.org/fulltext/{doi}
+    location_field: $.links.pdf
+    format_hint: pdf
+    headers:
+      Authorization: Bearer ${MYREPO_TOKEN}
+```
+
+To actually use a custom provider, add its name to the `full_text_providers`
+chain (it is not enabled merely by being defined), for example:
+
+```yaml
+full_text_providers_file: providers.yaml
+full_text_providers:
+  - myrepo
+  - pmc
+  - unpaywall
+  - openalex
+```
+
+## Provenance fields
+
+When a reference is enriched with full text, the cached reference records where
+the text came from. These provenance fields are written alongside the content:
+
+| Field | Description |
+|-------|-------------|
+| `full_text_provider` | Name of the provider that supplied the full text (e.g. `pmc`, `epmc_preprint`, `unpaywall`, or a custom name). |
+| `oa_status` | Open-access status reported by the provider (e.g. `gold`, `green`, `bronze`). |
+| `license` | License of the open-access copy, when known. |
+| `local_pdf_path` | Path (relative to the cache directory) to the downloaded PDF, when one was persisted. |
+
+When full text from a PDF is obtained, the reference's `content_type` becomes
+`full_text_pdf` and the extracted text replaces the abstract-only content. The
+downloaded PDF lives under the cache directory and can be located at
+`cache_dir / local_pdf_path`.
+
+## See Also
+
+- [Validating DOIs](validate-dois.md) — DOI metadata sources and supplementary files
+- [Validating Entrez Accessions](validate-entrez.md) — PubMed/PMC references
+- [CLI Reference](../reference/cli.md) — complete command documentation

{linkml_reference_validator-0.2.1rc1 → linkml_reference_validator-0.2.1rc3}/docs/reference/cli.md

@@ -761,6 +761,69 @@ linkml-reference-validator cache lookup PMID:16888623 --content
 linkml-reference-validator cache lookup PMID:16888623 --no-cache
 ```
 
+## cache export
+
+Export public bibliographic metadata to CSL JSON for Zotero import.
+
+```bash
+linkml-reference-validator cache export --output project-zotero.json [OPTIONS]
+```
+
+**Options:**
+
+- `--output PATH, -o PATH` - Required destination file
+- `--format TEXT` - Export format (default and currently supported: `csl-json`)
+- `--needs-full-text` - Export only publication records needing full text (default)
+- `--all` - Include eligible records that already contain full text
+- `--cache-dir PATH, -c PATH` - Public reference cache directory
+- `--force, -f` - Replace an existing output file
+- `--config PATH` - Validation configuration file
+- `--verbose, -v` - Enable detailed logging
+
+```bash
+linkml-reference-validator cache export \
+  --cache-dir references_cache \
+  --needs-full-text \
+  --output project-zotero.json
+```
+
+The output is a DOI/PMID-deduplicated metadata allowlist. It never includes
+cached article content, excerpts, PDFs, local paths, or private-cache data.
+
+---
+
+## cache enrich
+
+Inventory or enrich existing abstract-only cache entries through one full-text
+provider. This is primarily intended for opt-in private-library providers such
+as Zotero.
+
+```bash
+linkml-reference-validator cache enrich [OPTIONS]
+```
+
+**Options:**
+
+- `--provider TEXT` - Registered provider name (default: `zotero`)
+- `--cache-dir PATH, -c PATH` - Reference cache directory
+- `--private-cache-dir PATH` - Separate private research-cache destination (default: `~/.cache/linkml-reference-validator/private`)
+- `--dry-run` - Report matches without changing files (default)
+- `--apply` - Materialize usable matches into the private research cache
+- `--config PATH` - Validation configuration file
+- `--verbose, -v` - Enable detailed logging
+
+```bash
+# Safe inventory
+linkml-reference-validator cache enrich --provider zotero --dry-run
+
+# Apply reviewed exact matches
+linkml-reference-validator cache enrich --provider zotero --apply
+```
+
+The public source cache is never modified by this command, and validation never
+reads the private destination. Private-library content may be copyrighted; keep
+the research cache private.
+
 ---
 
 ## Reference ID Formats