linkml-reference-validator 0.2.1rc2__tar.gz → 0.3.0__tar.gz

{linkml_reference_validator-0.2.1rc2 → linkml_reference_validator-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: linkml_reference_validator
-Version: 0.2.1rc2
+Version: 0.3.0
 Summary: Validation of supporting text from references and publications
 Author-email: Chris Mungall <cjmungall@lbl.gov>
 License-Expression: Apache-2.0

{linkml_reference_validator-0.2.1rc2 → linkml_reference_validator-0.3.0}/docs/concepts/editorial-conventions.md

@@ -248,7 +248,7 @@ Editorial conventions work the same in data files:
 
 ### Normalization Order
 
-1. Remove everything inside `[...]`
+1. Remove editorial `[...]` content unless a configured literal pattern matches
 2. Split on `...` if present
 3. For each part:
    - Lowercase
@@ -258,10 +258,15 @@ Editorial conventions work the same in data files:
 
 ### Regular Expressions
 
-Brackets: `\[([^\]]*)\]` (removes content and brackets)
-Ellipsis: `\.\.\.` (splits on three dots exactly)
+Brackets: `\[.*?\]` (single-line, non-nested matching; literal patterns can preserve matches)
 
-Note: `..` (two dots) and `....` (four dots) are NOT treated as ellipsis.
+Ellipsis: `\s*\.{2,}\s*` (splits on two or more ASCII periods)
+
+Both `..` (two dots) and `....` (four dots) are treated as separators. The Unicode
+ellipsis `…` is not a separator.
+
+Python callers can use the public [supporting text splitting helper](../reference/splitting.md)
+without constructing a validator.
 
 ## See Also
 

{linkml_reference_validator-0.2.1rc2 → linkml_reference_validator-0.3.0}/docs/concepts/how-it-works.md

@@ -36,6 +36,51 @@ reference_content = "...The MUC1 oncoprotein blocks nuclear..."
 # Match: "muc1 oncoprotein" found in normalized reference
 ```
 
+**Empty excerpts are rejected:**
+
+An empty or whitespace-only excerpt is always an error, never a pass. The
+empty string is a substring of every document, so a blank excerpt would
+otherwise be reported as verified evidence while quoting nothing at all:
+
+```python
+supporting_text = ""     # or "   ", "\n"
+# ERROR: Supporting text is empty
+```
+
+The check runs before the reference is fetched, and applies even when the
+reference prefix is on the [skip list](../how-to/skip-unsupported-references.md):
+a blank excerpt is a defect in the data regardless of what it cites. An
+excerpt slot that is *absent* is a different matter — whether it is required
+is up to your schema.
+
+**Minimum excerpt length (optional):**
+
+A blank excerpt is the extreme case. A two-character one is barely better: a
+short string matches almost any paper by chance, so the match succeeds without
+the excerpt being real evidence. Set `min_excerpt_length` to require a minimum:
+
+```yaml
+# .linkml-reference-validator.yaml
+validation:
+  min_excerpt_length: 20
+```
+
+Keep it under `validation:`. A config file that has a `validation:` section
+ignores validation keys placed at the top level — silently, which for an
+opt-in check means it looks enabled while never running.
+
+The check is off by default (`0`), so existing data keeps validating as before.
+Length is measured in **non-whitespace characters of quoted text**:
+
+- Whitespace is ignored, so PDF-to-text extraction damage does not change the
+  count — `"t he pro tein"` and `"the protein"` both measure 10.
+- No tokenizing, so chemical nomenclature and other punctuation-dense strings
+  are counted in full — `"2,3-dihydroxybenzoate"` measures 21, not 2 words.
+- Editorial `[brackets]` and `...` separators do not count, since they are
+  never matched against the reference.
+
+Like the blank check, this runs before the reference is fetched.
+
 ### 3. Ellipsis Handling
 
 When supporting text contains `...`, each part is matched separately:

linkml_reference_validator-0.3.0/docs/how-to/export-cache-to-zotero.md

@@ -0,0 +1,114 @@
+# Export a Reference Cache to Zotero
+
+This workflow turns a checked-in public reference cache into a Zotero collection,
+asks Zotero to find available PDFs, and then inventories the resulting attachments
+without putting closed manuscripts into the project repository.
+
+The three stores remain separate:
+
+| Store | Purpose | Safe to commit? |
+|---|---|---:|
+| Project `references_cache` | Reproducible public validation evidence | Yes |
+| Zotero library | User-managed references and PDF attachments | No |
+| Private research cache | Extracted user-library text for agent context | No |
+
+Normal validation reads only the first store.
+
+## 1. Export metadata from the public cache
+
+Run the exporter from the `linkml-reference-validator` checkout. For dismech:
+
+```bash
+uv run linkml-reference-validator cache export \
+  --cache-dir /Users/cjm/repos/dismech/references_cache \
+  --needs-full-text \
+  --output /Users/cjm/Downloads/dismech-zotero.json
+```
+
+`--needs-full-text` is the default. It excludes cache entries that already have
+full text, excludes non-publication identifiers, and deduplicates by normalized
+DOI and then PMID. The CSL JSON output uses an explicit metadata allowlist:
+
+- title and authors;
+- journal and year;
+- DOI; or
+- PMID and its public PubMed URL.
+
+It never contains cached article text, excerpts, provenance, PDFs, local paths,
+or private-cache data. The command refuses to replace an existing output file;
+use `--force` only after checking the destination. Use `--all` if you explicitly
+want eligible references that already contain full text.
+
+## 2. Import the file into Zotero
+
+In Zotero:
+
+1. Choose **File → Import**.
+2. Choose **A file**.
+3. Select `dismech-zotero.json`.
+4. Keep the imported records in a project-specific collection such as
+   **dismech**.
+
+CSL JSON is a standard format supported by Zotero. Importing the file adds only
+bibliographic records; it does not copy anything from the validator's caches.
+
+## 3. Ask Zotero to find PDFs
+
+Select one or more top-level journal articles in the center item list,
+right-click the selection, and choose **Find Full Text**. It is a built-in Zotero
+command, not a plugin.
+
+![Zotero Find Full Text processing a batch of imported references](../assets/zotero-find-full-text.png)
+
+The command appears for an eligible parent item that has a DOI and does not
+already have a file attachment. It may also be unavailable in a group library
+where files cannot be added. If it is absent, first test a single article in
+your personal library and confirm that its DOI field is populated.
+
+For a large import, start with a modest batch rather than all records at once.
+Publishers can temporarily rate-limit repeated requests, and a later retry may
+find additional files. Zotero reports **Full Text PDF** or **No file found** for
+each attempted item.
+
+## 4. Inventory the enriched Zotero library
+
+Once Zotero finishes, scan the project cache again:
+
+```bash
+uv run linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir /Users/cjm/repos/dismech/references_cache \
+  --dry-run
+```
+
+This read-only scan matches exact DOI, PMID, or PMCID identifiers and reports
+which cached references now have usable Zotero text or PDF attachments. It does
+not invoke **Find Full Text** and does not modify Zotero.
+
+After reviewing the matches, materialize them into the separate private research
+cache:
+
+```bash
+uv run linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir /Users/cjm/repos/dismech/references_cache \
+  --apply
+```
+
+Apply mode leaves `references_cache` unchanged. Private content goes to
+`~/.cache/linkml-reference-validator/private` by default with owner-only
+permissions. Agents may inspect it for context, but validation cannot use it as
+evidence.
+
+## 5. Validate reproducibly
+
+Run validation normally. Results remain reproducible because the validator
+reads the public project cache and accepts only public full-text provider results.
+Neither the contents of Zotero nor the machine-specific private cache affect
+validation.
+
+## See also
+
+- [Fetching Full Text and PDFs](fetch-full-text-and-pdfs.md)
+- [Zotero: importing standardized formats](https://www.zotero.org/support/kb/importing_standardized_formats)
+- [Zotero: adding files to a library](https://www.zotero.org/support/attaching_files)

{linkml_reference_validator-0.2.1rc2 → linkml_reference_validator-0.3.0}/docs/how-to/fetch-full-text-and-pdfs.md

@@ -46,9 +46,11 @@ your config YAML or on the `ReferenceValidationConfig` object):
 |-----|---------|-------------|
 | `fetch_full_text` | `true` | Attempt to obtain full text via the provider chain when a metadata source does not already return full text. |
 | `full_text_providers` | `[pmc, epmc_preprint, unpaywall, openalex]` | Ordered list of provider names to try until one yields usable full text. |
+| `private_cache_dir` | `~/.cache/linkml-reference-validator/private` | Separate research cache for closed/user-library full text; never read by validation. |
 | `pdf_backend` | `pypdf` | Name of the PDF text-extraction backend. |
 | `download_pdfs` | `true` | If true, persist downloaded PDFs to the files cache directory. |
 | `full_text_providers_file` | `null` | Optional path to a YAML file defining custom full-text providers. |
+| `zotero_base_url` | `http://localhost:23119/api/users/0` | Read-only Zotero local API library used by the opt-in `zotero` provider. |
 
 Two existing keys are also reused by the full-text machinery:
 
@@ -71,6 +73,106 @@ pdf_backend: pypdf
 download_pdfs: true
 ```
 
+## Private manuscripts in Zotero
+
+The opt-in **`zotero`** provider can find non-open manuscripts that are already
+in your personal Zotero library. It matches exact DOI, PMID, or PMCID identifiers;
+it does not automatically accept fuzzy title matches.
+
+Zotero must be running, and its local API must be enabled in Zotero's advanced
+settings. Inventory the reference cache before changing it:
+
+```bash
+linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir references_cache \
+  --dry-run
+```
+
+The report distinguishes `found`, `not_found`, `already_full_text`, and `error`.
+Dry-run is the default and never changes cache files. After reviewing exact
+matches, apply usable text with:
+
+```bash
+linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir references_cache \
+  --apply
+```
+
+Apply mode does **not** modify `references_cache`. It writes the enriched entry
+to a separate research cache at `~/.cache/linkml-reference-validator/private`
+by default. Agents may inspect that cache for background context, but ordinary
+validation never reads it and never cites it as evidence. Validation therefore
+has the same inputs locally, in CI, and on another contributor's machine.
+
+To keep the research cache in a private repository or another secured location:
+
+```bash
+linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir references_cache \
+  --private-cache-dir /path/to/private-reference-cache \
+  --apply
+```
+
+The same location can be configured for all commands:
+
+```yaml
+cache_dir: references_cache
+private_cache_dir: /path/to/private-reference-cache
+```
+
+Zotero-indexed text is preferred. If Zotero has not indexed an attachment, the
+provider downloads it from Zotero's local PDF endpoint and sends it through the
+normal PDF extractor. The source attachment key and `user_library` access type
+are persisted, but the ephemeral localhost URL is not.
+
+!!! warning "Keep research caches private"
+
+    Private cache entries contain extracted manuscript text and may contain a
+    copied PDF. Do not place this cache in a public repository or share it
+    unless you have permission to redistribute that content.
+
+Do not add `zotero` to `full_text_providers`. Even if it is configured there,
+ordinary validation rejects its `user_library` result and continues to public
+providers. Use `cache enrich` when you want to search Zotero.
+
+Support for reproducible validation backed by private manuscripts is a future,
+separate feature. The proposed design is a trusted offline job that emits a
+minimal signed excerpt attestation into the public cache. The validator would
+verify the signature using a checked-in public key; neither agents nor CI would
+have the signing key or permission to generate those files directly.
+
+## Export a cache to a Zotero collection
+
+Export references that still need full text as metadata-only CSL JSON:
+
+```bash
+linkml-reference-validator cache export \
+  --cache-dir references_cache \
+  --format csl-json \
+  --needs-full-text \
+  --output project-zotero.json
+```
+
+The default `--needs-full-text` mode excludes cache entries that already contain
+full text. It also excludes non-publication identifiers, deduplicates by
+normalized DOI and then PMID, and exports only an explicit metadata allowlist:
+title, authors, journal, year, DOI, PMID, and PubMed URL. Cached article text,
+excerpts, provenance, PDFs, and local paths are never exported. Use `--all` only
+when you deliberately want eligible records that already have full text.
+
+The exporter refuses to replace an existing output file unless `--force` is
+given. Review the reported counts, then in Zotero choose **File → Import → A
+file**, select the JSON file, and place the imported items in a project-specific
+collection. Zotero supports CSL JSON as a standard import format. After import,
+select the collection and run **Find Full Text**, then rerun `cache enrich`.
+
+See [Zotero's standardized-format import instructions](https://www.zotero.org/support/kb/importing_standardized_formats).
+For the complete workflow with screenshots, see
+[Export a Reference Cache to Zotero](export-cache-to-zotero.md).
+
 ## Preprints
 
 Preprints are first-class references. They are increasingly cited for early
@@ -183,3 +285,46 @@ downloaded PDF lives under the cache directory and can be located at
 - [Validating DOIs](validate-dois.md) — DOI metadata sources and supplementary files
 - [Validating Entrez Accessions](validate-entrez.md) — PubMed/PMC references
 - [CLI Reference](../reference/cli.md) — complete command documentation
+
+## HTML article checks and cache recovery
+
+An OpenAlex or Unpaywall URL may lead to a repository item record containing
+only an abstract and metadata. HTML full-text acquisition now requires an
+identifiable research body: publisher body markup or narrative research sections
+with paragraphs, after removing abstracts and navigation. Length and additional
+metadata text alone do not establish full text. This also applies when a URL
+advertised as a PDF actually returns HTML. Genuine HTML research bodies remain
+eligible, including short bodies above the existing 500-character minimum.
+
+Ambiguous layouts are rejected and the next configured provider is tried. The
+fetcher does not crawl repository download links. If every provider completes
+without usable text or an error, the reference retains its original abstract
+and records `full_text_attempted: true`.
+
+New `full_text_html` cache entries carry `html_full_text_version: 1`. Older
+Markdown HTML full-text entries without this stamp are refreshed on their next
+validation read. This migration covers all providers because cached plain text
+cannot establish which HTML body structure was present. It does not invalidate
+current PDF, XML, or abstract caches.
+
+If refreshing an old HTML entry fails, validation does **not** fall back to that
+entry: its text may be repository metadata rather than article evidence. The
+file remains untouched for inventory/export, and the next validation run retries
+recovery. Once the source is available, normal fetching replaces it with a
+verified article or the available metadata/abstract. Other content types retain
+the existing stale-cache fallback. Legacy `.txt` caches remain a separate,
+read-only compatibility path and are not migrated by this HTML stamp.
+
+Research headings establish body presence; they do not filter the accepted
+article's other sections. For example, `Main`, `Case presentation`, and
+`Limitations` remain in an accepted article. A separately marked body is also
+preserved when an abstract heading uses a higher heading level. Without that
+structural boundary, an unmarked structured abstract remains ambiguous and is
+rejected conservatively.
+
+A provider's `FullTextLocation.text` (including a custom API `text_field`) is a
+trusted **already-extracted article body** contract. Configure it only for APIs
+that return article text, not arbitrary page text or metadata. Plain extracted
+text cannot be structurally rechecked; raw HTML supplied with an HTML hint still
+passes the body classifier. The acceptance stamp records that either the HTML
+check or this trusted provider contract was used.

{linkml_reference_validator-0.2.1rc2 → linkml_reference_validator-0.3.0}/docs/how-to/repair-validation-errors.md

@@ -13,6 +13,21 @@ After validating your data files, you may find validation errors due to:
 
 The `repair` command attempts to fix these issues automatically or flags them for manual review.
 
+### Typographic ligatures already match
+
+Validation folds Latin compatibility ligatures on both the reference content and
+the quote, including each part separated by `...`. For example, PDF-extracted
+`amyloid fibrils` matches `amyloid fibrils` without a repair. The supported folds
+are `ff → ff`, `fi → fi`, `fl → fl`, `ffi → ffi`, `ffl → ffl`, `ſt/st → st`, and
+`IJ/ij → IJ/ij`. IJ is included because Unicode gives it a compatibility
+decomposition into I and J; ordinary case normalization still applies.
+
+The distinct letters `Æ/æ` and `Œ/œ` remain distinct from `AE/ae` and `OE/oe`.
+Equating these letters would accept spelling changes beyond typographic repair.
+Validation uses an explicit ligature table instead of broad Unicode NFKC
+normalization so this change does not equate scientific notation such as `H₂O`
+with `H2O`, `10⁶` with `106`, or the micro sign `µ` with Greek `μ`.
+
 ## Quick Start
 
 ### Repair a Single Quote
@@ -112,7 +127,7 @@ Flags text that appears fabricated or hallucinated:
 
 ```
 RECOMMENDED REMOVALS:
-  PMID:34567890 at evidence[2]:
+  PMID:34567890 at evidence[2].supporting_text:
     Similarity: 8%
     Snippet: 'This completely made up text...'
 ```
@@ -290,6 +305,33 @@ The supporting text wasn't found in the reference. Possible causes:
 - Wrong PMID
 - AI-generated/hallucinated quote
 
+### "Supporting text is empty"
+
+The excerpt is empty or whitespace-only, so there is nothing to match against
+the reference. This is usually a bug in whatever generated the data (for
+example, a string slice with a backwards range yielding `""`).
+
+Empty excerpts cannot be repaired automatically — there is no text to correct,
+and fuzzy-matching nothing against the reference would invent a quote. They
+appear under **INSUFFICIENT EXCERPTS (nothing to verify)** in the report, so
+you can supply the real quote or drop the evidence item.
+
+They are deliberately *not* counted as removals. A removal recommendation means
+"this quote does not match the reference", a verdict reached by comparing the
+two; here no comparison happened. So `removal_count` excludes them and the
+action type is `INSUFFICIENT_EXCERPT`, not `REMOVAL` — worth knowing if you
+script against the report. They do still make `repair data` exit non-zero.
+
+### "Supporting text is too short"
+
+The excerpt is below the `min_excerpt_length` you configured (off by default,
+and set under `validation:`; see
+[How It Works](../concepts/how-it-works.md)). A very short excerpt matches
+almost any paper by chance, so it is weak evidence even when the match succeeds.
+
+Like empty excerpts, these report under **INSUFFICIENT EXCERPTS** rather than
+being repaired: there is no way to guess which longer passage the curator meant.
+
 ### "Reference content not available"
 
 The reference exists but has no retrievable content:

{linkml_reference_validator-0.2.1rc2 → linkml_reference_validator-0.3.0}/docs/how-to/use-local-files-and-urls.md

@@ -89,7 +89,7 @@ URLs are cached the same way as PMID and DOI references:
 
 ### Title Extraction
 
-For HTML pages, the title is extracted from the `<title>` tag. For other content types, the URL itself is used as the title.
+For HTML pages, the title is the `citation_title` meta tag when present, then the `<title>` tag. For PDFs, the validator tries a publisher landing page's `citation_title` and then the PDF's embedded title. When neither is found, and for other content types, the URL itself is used as the title. See [Validating URLs](validate-urls.md#titles).
 
 ### Example: Validating Against a Web Page
 
@@ -145,6 +145,6 @@ Both file and URL references work in LinkML data files:
 
 ## Limitations
 
-- **PDF files**: Not yet supported (planned for future)
+- **Local PDF files**: Not yet supported as `file:` references. PDFs fetched by `url:` are supported.
 - **Authentication**: URLs requiring login are not supported
 - **Dynamic content**: JavaScript-rendered pages may not work

{linkml_reference_validator-0.2.1rc2 → linkml_reference_validator-0.3.0}/docs/how-to/validate-clinical-trials.md

@@ -40,6 +40,10 @@ clinicaltrials:NCT00000001
 CLINICALTRIALS:NCT00000001
 ```
 
+These forms normalize to `clinicaltrials:NCT00000001` and share the cache file
+`clinicaltrials_NCT00000001.md`. Existing files with that canonical name are
+reused without renaming or fetching the trial again.
+
 ## Prefix Aliases and Normalization
 
 If your data uses alternate prefix styles (e.g., the legacy `NCT:` prefix), configure normalization in `.linkml-reference-validator.yaml`:

{linkml_reference_validator-0.2.1rc2 → linkml_reference_validator-0.3.0}/docs/how-to/validate-urls.md

@@ -15,8 +15,8 @@ The linkml-reference-validator supports validating references that point to web
 When a reference field contains a URL, the validator:
 
 1. Fetches the web page content
-2. Extracts the page title from `<title>` tag (for HTML)
-3. Caches the content for future validations
+2. Extracts the page title (see [Titles](#titles))
+3. Sanitizes HTML and caches the content for future validations
 4. Validates your supporting text against the page content
 
 ## URL Format
@@ -79,14 +79,45 @@ When the validator encounters a URL reference, it:
 
 The fetcher stores:
 
-- **Title**: Extracted from the `<title>` tag (for HTML pages)
-- **Content**: The raw page content as received
-- **Content type**: Marked as `url` to distinguish from other reference types
-
-Note: The validator stores raw page content without HTML-to-text conversion.
-HTML tags remain in the cached file, and tag names can surface during
-normalization. If validation fails because tags interrupt the text, consider
-extracting plain text and validating against a local `file:` reference instead.
+- **Title**: See [Titles](#titles)
+- **Content**: Sanitized HTML for HTML pages; plain text and XML as received;
+  extracted text for PDFs
+- **Content type**: `url` for pages, `full_text_pdf` for PDFs with text
+
+HTML is sanitized before it is cached, because caches are often committed to
+public repositories and page scripts routinely carry signed asset URLs and API
+keys. `<script>`, `<style>`, `<noscript>` and `<template>` elements and HTML
+comments are removed. So are `<meta>`, `<link>` and `<base>`, which hold nothing
+but attributes. Every other tag attribute is removed except `rowspan`,
+`colspan` and `scope`, which carry table structure. Body markup and text are
+kept. There is no HTML-to-text conversion, so tag names can still surface
+during normalization. If validation fails because tags interrupt the text,
+consider extracting plain text and validating against a local `file:`
+reference instead.
+
+#### Titles
+
+For HTML pages the title is the `citation_title` meta tag when present (the
+Highwire / Google Scholar convention, which names the article rather than the
+site), then the `<title>` tag.
+
+For PDFs the validator tries, in order:
+
+1. The `citation_title` of the publisher's landing page, where a known rule
+   maps the PDF URL to it. Currently J-STAGE: `.../_pdf` becomes
+   `.../_article`.
+2. The PDF's embedded `/Title` metadata, ignoring placeholders such as
+   `Microsoft Word - draft.doc`, bare filenames and `Untitled`.
+3. The URL itself.
+
+So a PDF entry whose `title` equals its URL is one for which no title was
+found.
+
+The landing-page rules and the placeholder titles are kept in
+`src/linkml_reference_validator/etl/rules.py`, with the other publisher and
+site rules. Each rule sits beside examples of what it must and must not match.
+To support another publisher, add a rule there with at least one example;
+`tests/test_rules.py` fails for a rule without one.
 
 ### 3. Caching
 
@@ -104,9 +135,9 @@ content_type: url
 ## Content
 
 <html>
-  <head>
-    <title>Chapter 3: Cell Structure and Function</title>
-  </head>
+<head>
+<title>Chapter 3: Cell Structure and Function</title>
+</head>
   ...
 ```
 
@@ -145,9 +176,10 @@ URL validation is designed for static web pages. It may not work well with:
 
 ### Raw Content
 
-The validator stores raw page content. For HTML pages:
+The validator stores sanitized page markup, not extracted text. For HTML pages:
 
-- HTML tags are preserved in the cache
+- HTML tags are preserved in the cache, without attributes other than
+  `rowspan`, `colspan` and `scope`
 - The text normalization during validation handles most cases
 - Complex HTML layouts may require careful text extraction