linkml-reference-validator 0.2.1rc3__tar.gz → 0.3.0__tar.gz

{linkml_reference_validator-0.2.1rc3 → linkml_reference_validator-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: linkml_reference_validator
-Version: 0.2.1rc3
+Version: 0.3.0
 Summary: Validation of supporting text from references and publications
 Author-email: Chris Mungall <cjmungall@lbl.gov>
 License-Expression: Apache-2.0

{linkml_reference_validator-0.2.1rc3 → linkml_reference_validator-0.3.0}/docs/concepts/editorial-conventions.md

@@ -248,7 +248,7 @@ Editorial conventions work the same in data files:
 
 ### Normalization Order
 
-1. Remove everything inside `[...]`
+1. Remove editorial `[...]` content unless a configured literal pattern matches
 2. Split on `...` if present
 3. For each part:
    - Lowercase
@@ -258,10 +258,15 @@ Editorial conventions work the same in data files:
 
 ### Regular Expressions
 
-Brackets: `\[([^\]]*)\]` (removes content and brackets)
-Ellipsis: `\.\.\.` (splits on three dots exactly)
+Brackets: `\[.*?\]` (single-line, non-nested matching; literal patterns can preserve matches)
 
-Note: `..` (two dots) and `....` (four dots) are NOT treated as ellipsis.
+Ellipsis: `\s*\.{2,}\s*` (splits on two or more ASCII periods)
+
+Both `..` (two dots) and `....` (four dots) are treated as separators. The Unicode
+ellipsis `…` is not a separator.
+
+Python callers can use the public [supporting text splitting helper](../reference/splitting.md)
+without constructing a validator.
 
 ## See Also
 

{linkml_reference_validator-0.2.1rc3 → linkml_reference_validator-0.3.0}/docs/concepts/how-it-works.md

@@ -36,6 +36,51 @@ reference_content = "...The MUC1 oncoprotein blocks nuclear..."
 # Match: "muc1 oncoprotein" found in normalized reference
 ```
 
+**Empty excerpts are rejected:**
+
+An empty or whitespace-only excerpt is always an error, never a pass. The
+empty string is a substring of every document, so a blank excerpt would
+otherwise be reported as verified evidence while quoting nothing at all:
+
+```python
+supporting_text = ""     # or "   ", "\n"
+# ERROR: Supporting text is empty
+```
+
+The check runs before the reference is fetched, and applies even when the
+reference prefix is on the [skip list](../how-to/skip-unsupported-references.md):
+a blank excerpt is a defect in the data regardless of what it cites. An
+excerpt slot that is *absent* is a different matter — whether it is required
+is up to your schema.
+
+**Minimum excerpt length (optional):**
+
+A blank excerpt is the extreme case. A two-character one is barely better: a
+short string matches almost any paper by chance, so the match succeeds without
+the excerpt being real evidence. Set `min_excerpt_length` to require a minimum:
+
+```yaml
+# .linkml-reference-validator.yaml
+validation:
+  min_excerpt_length: 20
+```
+
+Keep it under `validation:`. A config file that has a `validation:` section
+ignores validation keys placed at the top level — silently, which for an
+opt-in check means it looks enabled while never running.
+
+The check is off by default (`0`), so existing data keeps validating as before.
+Length is measured in **non-whitespace characters of quoted text**:
+
+- Whitespace is ignored, so PDF-to-text extraction damage does not change the
+  count — `"t he pro tein"` and `"the protein"` both measure 10.
+- No tokenizing, so chemical nomenclature and other punctuation-dense strings
+  are counted in full — `"2,3-dihydroxybenzoate"` measures 21, not 2 words.
+- Editorial `[brackets]` and `...` separators do not count, since they are
+  never matched against the reference.
+
+Like the blank check, this runs before the reference is fetched.
+
 ### 3. Ellipsis Handling
 
 When supporting text contains `...`, each part is matched separately:

{linkml_reference_validator-0.2.1rc3 → linkml_reference_validator-0.3.0}/docs/how-to/fetch-full-text-and-pdfs.md

@@ -285,3 +285,46 @@ downloaded PDF lives under the cache directory and can be located at
 - [Validating DOIs](validate-dois.md) — DOI metadata sources and supplementary files
 - [Validating Entrez Accessions](validate-entrez.md) — PubMed/PMC references
 - [CLI Reference](../reference/cli.md) — complete command documentation
+
+## HTML article checks and cache recovery
+
+An OpenAlex or Unpaywall URL may lead to a repository item record containing
+only an abstract and metadata. HTML full-text acquisition now requires an
+identifiable research body: publisher body markup or narrative research sections
+with paragraphs, after removing abstracts and navigation. Length and additional
+metadata text alone do not establish full text. This also applies when a URL
+advertised as a PDF actually returns HTML. Genuine HTML research bodies remain
+eligible, including short bodies above the existing 500-character minimum.
+
+Ambiguous layouts are rejected and the next configured provider is tried. The
+fetcher does not crawl repository download links. If every provider completes
+without usable text or an error, the reference retains its original abstract
+and records `full_text_attempted: true`.
+
+New `full_text_html` cache entries carry `html_full_text_version: 1`. Older
+Markdown HTML full-text entries without this stamp are refreshed on their next
+validation read. This migration covers all providers because cached plain text
+cannot establish which HTML body structure was present. It does not invalidate
+current PDF, XML, or abstract caches.
+
+If refreshing an old HTML entry fails, validation does **not** fall back to that
+entry: its text may be repository metadata rather than article evidence. The
+file remains untouched for inventory/export, and the next validation run retries
+recovery. Once the source is available, normal fetching replaces it with a
+verified article or the available metadata/abstract. Other content types retain
+the existing stale-cache fallback. Legacy `.txt` caches remain a separate,
+read-only compatibility path and are not migrated by this HTML stamp.
+
+Research headings establish body presence; they do not filter the accepted
+article's other sections. For example, `Main`, `Case presentation`, and
+`Limitations` remain in an accepted article. A separately marked body is also
+preserved when an abstract heading uses a higher heading level. Without that
+structural boundary, an unmarked structured abstract remains ambiguous and is
+rejected conservatively.
+
+A provider's `FullTextLocation.text` (including a custom API `text_field`) is a
+trusted **already-extracted article body** contract. Configure it only for APIs
+that return article text, not arbitrary page text or metadata. Plain extracted
+text cannot be structurally rechecked; raw HTML supplied with an HTML hint still
+passes the body classifier. The acceptance stamp records that either the HTML
+check or this trusted provider contract was used.

{linkml_reference_validator-0.2.1rc3 → linkml_reference_validator-0.3.0}/docs/how-to/repair-validation-errors.md

@@ -13,6 +13,21 @@ After validating your data files, you may find validation errors due to:
 
 The `repair` command attempts to fix these issues automatically or flags them for manual review.
 
+### Typographic ligatures already match
+
+Validation folds Latin compatibility ligatures on both the reference content and
+the quote, including each part separated by `...`. For example, PDF-extracted
+`amyloid fibrils` matches `amyloid fibrils` without a repair. The supported folds
+are `ff → ff`, `fi → fi`, `fl → fl`, `ffi → ffi`, `ffl → ffl`, `ſt/st → st`, and
+`IJ/ij → IJ/ij`. IJ is included because Unicode gives it a compatibility
+decomposition into I and J; ordinary case normalization still applies.
+
+The distinct letters `Æ/æ` and `Œ/œ` remain distinct from `AE/ae` and `OE/oe`.
+Equating these letters would accept spelling changes beyond typographic repair.
+Validation uses an explicit ligature table instead of broad Unicode NFKC
+normalization so this change does not equate scientific notation such as `H₂O`
+with `H2O`, `10⁶` with `106`, or the micro sign `µ` with Greek `μ`.
+
 ## Quick Start
 
 ### Repair a Single Quote
@@ -112,7 +127,7 @@ Flags text that appears fabricated or hallucinated:
 
 ```
 RECOMMENDED REMOVALS:
-  PMID:34567890 at evidence[2]:
+  PMID:34567890 at evidence[2].supporting_text:
     Similarity: 8%
     Snippet: 'This completely made up text...'
 ```
@@ -290,6 +305,33 @@ The supporting text wasn't found in the reference. Possible causes:
 - Wrong PMID
 - AI-generated/hallucinated quote
 
+### "Supporting text is empty"
+
+The excerpt is empty or whitespace-only, so there is nothing to match against
+the reference. This is usually a bug in whatever generated the data (for
+example, a string slice with a backwards range yielding `""`).
+
+Empty excerpts cannot be repaired automatically — there is no text to correct,
+and fuzzy-matching nothing against the reference would invent a quote. They
+appear under **INSUFFICIENT EXCERPTS (nothing to verify)** in the report, so
+you can supply the real quote or drop the evidence item.
+
+They are deliberately *not* counted as removals. A removal recommendation means
+"this quote does not match the reference", a verdict reached by comparing the
+two; here no comparison happened. So `removal_count` excludes them and the
+action type is `INSUFFICIENT_EXCERPT`, not `REMOVAL` — worth knowing if you
+script against the report. They do still make `repair data` exit non-zero.
+
+### "Supporting text is too short"
+
+The excerpt is below the `min_excerpt_length` you configured (off by default,
+and set under `validation:`; see
+[How It Works](../concepts/how-it-works.md)). A very short excerpt matches
+almost any paper by chance, so it is weak evidence even when the match succeeds.
+
+Like empty excerpts, these report under **INSUFFICIENT EXCERPTS** rather than
+being repaired: there is no way to guess which longer passage the curator meant.
+
 ### "Reference content not available"
 
 The reference exists but has no retrievable content:

{linkml_reference_validator-0.2.1rc3 → linkml_reference_validator-0.3.0}/docs/how-to/use-local-files-and-urls.md

@@ -89,7 +89,7 @@ URLs are cached the same way as PMID and DOI references:
 
 ### Title Extraction
 
-For HTML pages, the title is extracted from the `<title>` tag. For other content types, the URL itself is used as the title.
+For HTML pages, the title is the `citation_title` meta tag when present, then the `<title>` tag. For PDFs, the validator tries a publisher landing page's `citation_title` and then the PDF's embedded title. When neither is found, and for other content types, the URL itself is used as the title. See [Validating URLs](validate-urls.md#titles).
 
 ### Example: Validating Against a Web Page
 
@@ -145,6 +145,6 @@ Both file and URL references work in LinkML data files:
 
 ## Limitations
 
-- **PDF files**: Not yet supported (planned for future)
+- **Local PDF files**: Not yet supported as `file:` references. PDFs fetched by `url:` are supported.
 - **Authentication**: URLs requiring login are not supported
 - **Dynamic content**: JavaScript-rendered pages may not work

{linkml_reference_validator-0.2.1rc3 → linkml_reference_validator-0.3.0}/docs/how-to/validate-clinical-trials.md

@@ -40,6 +40,10 @@ clinicaltrials:NCT00000001
 CLINICALTRIALS:NCT00000001
 ```
 
+These forms normalize to `clinicaltrials:NCT00000001` and share the cache file
+`clinicaltrials_NCT00000001.md`. Existing files with that canonical name are
+reused without renaming or fetching the trial again.
+
 ## Prefix Aliases and Normalization
 
 If your data uses alternate prefix styles (e.g., the legacy `NCT:` prefix), configure normalization in `.linkml-reference-validator.yaml`:

{linkml_reference_validator-0.2.1rc3 → linkml_reference_validator-0.3.0}/docs/how-to/validate-urls.md

@@ -15,8 +15,8 @@ The linkml-reference-validator supports validating references that point to web
 When a reference field contains a URL, the validator:
 
 1. Fetches the web page content
-2. Extracts the page title from `<title>` tag (for HTML)
-3. Caches the content for future validations
+2. Extracts the page title (see [Titles](#titles))
+3. Sanitizes HTML and caches the content for future validations
 4. Validates your supporting text against the page content
 
 ## URL Format
@@ -79,14 +79,45 @@ When the validator encounters a URL reference, it:
 
 The fetcher stores:
 
-- **Title**: Extracted from the `<title>` tag (for HTML pages)
-- **Content**: The raw page content as received
-- **Content type**: Marked as `url` to distinguish from other reference types
-
-Note: The validator stores raw page content without HTML-to-text conversion.
-HTML tags remain in the cached file, and tag names can surface during
-normalization. If validation fails because tags interrupt the text, consider
-extracting plain text and validating against a local `file:` reference instead.
+- **Title**: See [Titles](#titles)
+- **Content**: Sanitized HTML for HTML pages; plain text and XML as received;
+  extracted text for PDFs
+- **Content type**: `url` for pages, `full_text_pdf` for PDFs with text
+
+HTML is sanitized before it is cached, because caches are often committed to
+public repositories and page scripts routinely carry signed asset URLs and API
+keys. `<script>`, `<style>`, `<noscript>` and `<template>` elements and HTML
+comments are removed. So are `<meta>`, `<link>` and `<base>`, which hold nothing
+but attributes. Every other tag attribute is removed except `rowspan`,
+`colspan` and `scope`, which carry table structure. Body markup and text are
+kept. There is no HTML-to-text conversion, so tag names can still surface
+during normalization. If validation fails because tags interrupt the text,
+consider extracting plain text and validating against a local `file:`
+reference instead.
+
+#### Titles
+
+For HTML pages the title is the `citation_title` meta tag when present (the
+Highwire / Google Scholar convention, which names the article rather than the
+site), then the `<title>` tag.
+
+For PDFs the validator tries, in order:
+
+1. The `citation_title` of the publisher's landing page, where a known rule
+   maps the PDF URL to it. Currently J-STAGE: `.../_pdf` becomes
+   `.../_article`.
+2. The PDF's embedded `/Title` metadata, ignoring placeholders such as
+   `Microsoft Word - draft.doc`, bare filenames and `Untitled`.
+3. The URL itself.
+
+So a PDF entry whose `title` equals its URL is one for which no title was
+found.
+
+The landing-page rules and the placeholder titles are kept in
+`src/linkml_reference_validator/etl/rules.py`, with the other publisher and
+site rules. Each rule sits beside examples of what it must and must not match.
+To support another publisher, add a rule there with at least one example;
+`tests/test_rules.py` fails for a rule without one.
 
 ### 3. Caching
 
@@ -104,9 +135,9 @@ content_type: url
 ## Content
 
 <html>
-  <head>
-    <title>Chapter 3: Cell Structure and Function</title>
-  </head>
+<head>
+<title>Chapter 3: Cell Structure and Function</title>
+</head>
   ...
 ```
 
@@ -145,9 +176,10 @@ URL validation is designed for static web pages. It may not work well with:
 
 ### Raw Content
 
-The validator stores raw page content. For HTML pages:
+The validator stores sanitized page markup, not extracted text. For HTML pages:
 
-- HTML tags are preserved in the cache
+- HTML tags are preserved in the cache, without attributes other than
+  `rowspan`, `colspan` and `scope`
 - The text normalization during validation handles most cases
 - Complex HTML layouts may require careful text extraction
 

{linkml_reference_validator-0.2.1rc3 → linkml_reference_validator-0.3.0}/docs/reference/cli.md

@@ -378,29 +378,57 @@ linkml-reference-validator validate data \
 
 ### Output Format
 
-**Success:**
-```
-Validating data.yaml against schema schema.yaml
-Cache directory: references_cache
-
-Validation Summary:
-  Total checks: 3
-  All validations passed!
-```
-
-**Failure:**
-```
-Validating data.yaml against schema schema.yaml
-Cache directory: references_cache
-
-Validation Issues (2):
-  [ERROR] Text part not found as substring: 'MUC1 activates JAK-STAT'
-    Location: Statement
+The summary separates work performed from issues found:
 
+```text
 Validation Summary:
-  Total checks: 3
-  Issues found: 2
-```
+  Input files: 2
+  Files validated: 2
+  Snippets checked: 3
+  Snippets skipped: 0
+  Snippets unavailable: 0
+  Titles checked: 1
+  Issues found: 1
+  Issues found in 1 file(s) (1 validation issue(s))
+  Failing files:
+    bad.yaml
+```
+
+- **Input files** counts supplied paths, including files that cannot be read.
+- **Files validated** counts files with at least one mapping submitted to the
+  validator. An empty mapping counts; an empty list has no instances and does not.
+  A mixed list containing valid mappings and malformed entries counts once and
+  still fails. Entry diagnostics use zero-based indexes.
+- **Snippets checked** counts executed comparisons against reference content,
+  including matches and mismatches. Multiple excerpt slots each count separately.
+- **Snippets skipped** counts snippet/reference pairs bypassed by `skip_prefixes`.
+- **Snippets unavailable** counts attempted snippet validations where the reference
+  could not be fetched or had no content. These are not completed comparisons.
+- **Titles checked** counts actual title comparisons, including those performed
+  alongside snippets. Missing titles and skipped/unavailable references do not count.
+- **Issues found** counts emitted validation issues. Blank excerpts may produce an
+  issue without any comparison; file read/parse or data-format errors appear in the
+  failing-file list without adding a fabricated validation issue.
+
+Absent excerpts and excerpts without a usable reference perform no comparison.
+Counters come from execution and reset for each LinkML validation call; the CLI
+accumulates all instances and files. The public plugin exposes `snippets_checked`,
+`snippets_skipped`, `snippets_unavailable`, and `titles_checked` after validation.
+When using lazy `process()` or `iter_results()` APIs, exhaust the result iterator
+before reading final counters. Unconsumed or partially consumed iterators do not
+provide final counts; `Validator.validate()` consumes the iterator for you.
+
+Title-only skips and unavailable titles have no separate summary counters: they
+leave all check counters at zero. A skipped title emits no issue; an unavailable
+standalone title emits an issue. Existing title behavior is preserved: a reference
+missing its title is an issue for standalone title validation, but a title supplied
+alongside an excerpt is not compared or reported when the reference has no title.
+
+A zero-snippet run prints `No snippet comparisons were performed.` This includes
+empty input collections and title-only runs. Exit behavior is unchanged: no issues
+or file errors means exit 0, even when zero snippets were checked; issues or file
+errors mean exit 1. Failed file paths are listed at the end, including unreadable
+files, while later files continue to be processed.
 
 ---
 
@@ -585,16 +613,16 @@ Repair Report
 ============================================================
 
 HIGH CONFIDENCE FIXES (auto-applicable):
-  PMID:12345678 at evidence[0]:
+  PMID:12345678 at evidence[0].supporting_text:
     Character normalization fix
     'CO2 levels...' → 'CO₂ levels...'
 
 SUGGESTED FIXES (review recommended):
-  PMID:23456789 at evidence[1]:
+  PMID:23456789 at evidence[1].supporting_text:
     Inserted ellipsis between non-contiguous parts
 
 RECOMMENDED REMOVALS (low confidence):
-  PMID:34567890 at evidence[2]:
+  PMID:34567890 at evidence[2].supporting_text:
     Similarity: 8%
     Snippet: 'Fabricated text...'
 
@@ -622,6 +650,10 @@ validation:
     geo: GEO
     NCBIGeo: GEO
 
+  # Minimum non-whitespace characters an excerpt must quote.
+  # 0 disables the check; empty excerpts are rejected regardless.
+  min_excerpt_length: 0
+
 repair:
   # Confidence thresholds
   auto_fix_threshold: 0.95
@@ -719,6 +751,31 @@ Successfully cached PMID:16888623
   Content length: 1523 characters
 ```
 
+### Exit Codes
+
+- `0` - The public validation cache holds a current entry for the reference
+- `1` - It does not
+
+Exit `0` does not imply a download: a reference the current extractor has
+already cached may be reported as cached without contacting the source.
+
+Two cases satisfy the promise loosely. Content marked as non-open full text is
+written to the private cache, which validation deliberately never reads, so that
+run exits `0` without leaving anything in the public cache. Legacy `.txt` entries
+are exempt from the extractor version check and so always report as current.
+
+Conversely, a reference that could not be re-fetched is a failure even when a
+cache entry for it exists. Validation falls back to an entry written by an older
+extractor rather than reporting the reference as missing, but this command exists
+to populate the cache, so leaving it without a current entry is reported as
+failure and a script gating on the exit status does not go green through an
+outage. The same applies when no source handles the identifier at all.
+
+```
+Fetching PMID:16888623...
+Failed to cache PMID:16888623: it could not be re-fetched, so an out-of-date cache entry was served. The cache still holds no current entry for it.
+```
+
 ---
 
 ## cache lookup

linkml_reference_validator-0.3.0/docs/reference/splitting.md

@@ -0,0 +1,69 @@
+# Splitting supporting text in Python
+
+The public import is:
+
+```python
+from linkml_reference_validator.matching import split_supporting_text
+
+parts = split_supporting_text("protein [important] functions ... in cells")
+assert parts == ["protein functions", "in cells"]
+```
+
+This pure helper needs no validator, fetcher, cache, or network access. The
+validator's existing `_split_query` method delegates to it, so offline snippet
+reports and validation share the same splitting implementation.
+
+## Contract
+
+`split_supporting_text(text, literal_bracket_patterns=()) -> list[str]`
+
+- Replace editorial bracketed text with a space.
+- Split on runs of **two or more ASCII periods**, including `..`, `...`, and
+  `....`. A single period and the Unicode ellipsis `…` remain literal.
+- Collapse whitespace within each part, trim it, and discard empty parts.
+- Return parts in input order; empty text or only editorial notes/separators
+  produces `[]`.
+
+The helper performs splitting only; it does not normalize case or punctuation,
+check a reference, or enforce validation policies such as minimum excerpt length.
+An empty list contains no quoted evidence and should not be treated as a verified
+snippet.
+
+Bracket parsing retains the existing non-nested, single-line regex behavior:
+`[` matches through the next `]` on the same line. Unmatched brackets remain.
+Periods inside preserved brackets still act as separators.
+
+## Literal scientific notation and configuration
+
+The default is an empty pattern sequence, matching
+`ReferenceValidationConfig.literal_bracket_patterns`. **No scientific notation is
+preserved automatically.** Each supplied regex is searched against the content
+inside brackets, without the brackets themselves. Any match preserves the whole
+bracketed text; otherwise it is removed. Supplied patterns are the complete set
+of exceptions, with no implicit patterns added.
+
+```python
+from linkml_reference_validator.models import ReferenceValidationConfig
+
+config = ReferenceValidationConfig(literal_bracket_patterns=[r"\d", r"[()+]"])
+parts = split_supporting_text(
+    "binds [2Fe-2S] [important] ... exports [poly(A)+] RNA",
+    literal_bracket_patterns=config.literal_bracket_patterns,
+)
+assert parts == ["binds [2Fe-2S]", "exports [poly(A)+] RNA"]
+```
+
+For repeated calls, pass a sequence of compiled string regexes to reuse them.
+Compiled flags are preserved; invalid regex strings raise `re.error`.
+
+```python
+import re
+
+patterns = tuple(re.compile(p) for p in config.literal_bracket_patterns)
+assert split_supporting_text("[2Fe-2S] ... [poly(A)+]", patterns) == [
+    "[2Fe-2S]", "[poly(A)+]"
+]
+```
+
+See [Editorial Conventions](../concepts/editorial-conventions.md) for authoring
+examples.

{linkml_reference_validator-0.2.1rc3 → linkml_reference_validator-0.3.0}/docs/setup-guide.md

@@ -385,6 +385,10 @@ validation:
   # Cache directory (relative to config file or absolute)
   cache_dir: ./references_cache
 
+  # Reject excerpts quoting fewer than this many non-whitespace
+  # characters (0 disables; empty excerpts are always rejected)
+  min_excerpt_length: 0
+
   # Custom reference prefix mappings
   reference_prefix_map:
     geo: GEO