linkml-reference-validator 0.3.1rc1__tar.gz → 0.5.0__tar.gz

{linkml_reference_validator-0.3.1rc1 → linkml_reference_validator-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: linkml_reference_validator
-Version: 0.3.1rc1
+Version: 0.5.0
 Summary: Validation of supporting text from references and publications
 Author-email: Chris Mungall <cjmungall@lbl.gov>
 License-Expression: Apache-2.0
@@ -58,6 +58,28 @@ linkml-validate -s schema.yaml data.yaml \
 
 ---
 
+## Agent skill
+
+Install the [linkml-reference-validator workflow skill](skills/linkml-reference-validator/SKILL.md) with the
+[skills CLI](https://skills.sh/) (requires Node.js). Preview available skills first:
+
+```bash
+npx skills add linkml/linkml-reference-validator --list
+npx skills add linkml/linkml-reference-validator --skill linkml-reference-validator
+```
+
+Installation defaults to the current project. Use `-a codex` or
+`-a claude-code` to select an agent; add `-g` for your user-wide skills directory:
+
+```bash
+npx skills add linkml/linkml-reference-validator --skill linkml-reference-validator -a codex -g
+```
+
+The skill helps agents configure reference QC, interpret failures and coverage,
+and set up repository hooks and CI using bundled reference guides. Install the
+runtime separately as described above; installing the skill does not activate
+hooks or configure CI, credentials, or data sources.
+
 ## Why Use This Tool?
 
 Scientific data often includes claims supported by quotes from publications. But how do you know the quotes are accurate?

{linkml_reference_validator-0.3.1rc1 → linkml_reference_validator-0.5.0}/README.md

@@ -37,6 +37,28 @@ linkml-validate -s schema.yaml data.yaml \
 
 ---
 
+## Agent skill
+
+Install the [linkml-reference-validator workflow skill](skills/linkml-reference-validator/SKILL.md) with the
+[skills CLI](https://skills.sh/) (requires Node.js). Preview available skills first:
+
+```bash
+npx skills add linkml/linkml-reference-validator --list
+npx skills add linkml/linkml-reference-validator --skill linkml-reference-validator
+```
+
+Installation defaults to the current project. Use `-a codex` or
+`-a claude-code` to select an agent; add `-g` for your user-wide skills directory:
+
+```bash
+npx skills add linkml/linkml-reference-validator --skill linkml-reference-validator -a codex -g
+```
+
+The skill helps agents configure reference QC, interpret failures and coverage,
+and set up repository hooks and CI using bundled reference guides. Install the
+runtime separately as described above; installing the skill does not activate
+hooks or configure CI, credentials, or data sources.
+
 ## Why Use This Tool?
 
 Scientific data often includes claims supported by quotes from publications. But how do you know the quotes are accurate?

{linkml_reference_validator-0.3.1rc1 → linkml_reference_validator-0.5.0}/docs/concepts/content-types.md

@@ -12,6 +12,7 @@ When fetching references, the validator tracks what type of content was retrieve
 | `summary` | GEO, BioProject, BioSample, ClinicalTrials | Database summary/description field |
 | `local_file` | file: | Content read from local filesystem |
 | `local_svg` | file: (`.svg`) | Text an SVG figure renders, a blank line between labels; a quote must fall within one label |
+| `local_drawio` | file: (`.drawio`) | Labels a drawio diagram draws, read like `local_svg` |
 | `url` | url: | Content fetched from web URL |
 | `unavailable` | Any | No content could be retrieved |
 

{linkml_reference_validator-0.3.1rc1 → linkml_reference_validator-0.5.0}/docs/concepts/how-it-works.md

@@ -13,7 +13,23 @@ linkml-reference-validator validates that quoted text (supporting text) actually
 Before matching, both the supporting text and reference content are normalized:
 
 - **Lowercased**: `"MUC1"` → `"muc1"`
-- **Punctuation removed**: `"c-Abl"` → `"c abl"`
+- **Punctuation removed**: `"c-Abl"` → `"c abl"` (a point or comma between
+  digits is kept, so `"10.5"` stays one number)
+- **Meaning-bearing symbols kept as words**: `<` `>` `≤` `≥` become `lt` `gt`
+  `le` `ge`, `±` (or `+/-`) becomes `pm`, and a minus sign before a number stays
+  on it, so `"p < 0.05"` does not match `"p > 0.05"` and `"2.5 fold"` does not
+  match `"-2.5 fold"`. ASCII and Unicode spellings are equivalent (`<=` and `≤`,
+  `-` and `−`). Publisher spellings count too: MEDLINE's `>/=` and `</=` for
+  `≥` and `≤`, a middle-dot decimal (`57·4` is `57.4`), and an en dash or
+  Unicode hyphen printed as a minus sign (`z score, –1.98`) or inside `+/–`.
+  A `−` set off from its number by a space (`from − 2.92`) is still a sign,
+  unless a number comes before it (`10 − 2` is a subtraction). A hyphen or en
+  dash after a word or digit (`IL-6`, `3-5`, `1–12`) is still punctuation. A
+  dash after a space and before a digit is a sign, so a range spaced on one
+  side only (`18 –65`) reads as `18` and `-65`; quote it as the source
+  prints it. A middle dot after a letter-joined number is not a decimal
+  point, so `MgCl2·6H2O` keeps its hydrate dot (and `pH7·4` does not read as
+  `pH7.4`; `pH 7·4` does).
 - **Whitespace collapsed**: Multiple spaces become single space
 - **Editorial notes removed**: `"[mucin 1]"` → `""`
 
@@ -36,6 +52,34 @@ reference_content = "...The MUC1 oncoprotein blocks nuclear..."
 # Match: "muc1 oncoprotein" found in normalized reference
 ```
 
+**Matches start and end on whole words:**
+
+A quote may not begin or end inside a word of the reference. Otherwise a quote
+stating the opposite of the paper would pass:
+
+```python
+supporting_text = "activates AKT signaling"
+reference_content = "...Loss of PTEN inactivates AKT signaling..."
+# NOT FOUND: "activates" only occurs inside "inactivates"
+
+supporting_text = "5 mM H2O2"
+reference_content = "...treated with 10.5 mM H2O2..."
+# NOT FOUND: "5" only occurs inside "10.5"
+```
+
+Each part of a quote split with `...` is held to the same rule.
+
+One allowance: when a quote ends in a letter, the reference word it ends on may
+carry trailing digits. Full-text extraction welds superscript citations onto the
+word before them, so `tumor suppressor<sup>12</sup>` reads as `suppressor12`,
+and a quote of `PTEN is a tumor suppressor` must still match. A quote ending in
+a digit gets no allowance (`10` never matches `105`). The same allowance means a
+quote ending in `AKT` matches `AKT1`, as plain substring matching always did.
+
+Because a point or comma between digits stays part of the number, numbers must
+be quoted with the reference's own separators: `days 3, 5 and 7` does not match
+`days 3,5 and 7`, and `10 000` does not match `10,000`.
+
 **Empty excerpts are rejected:**
 
 An empty or whitespace-only excerpt is always an error, never a pass. The
@@ -224,6 +268,16 @@ content_type: abstract_only
 The MUC1 oncoprotein blocks nuclear targeting...
 ```
 
+An entry with no title has no `# title`, authors or journal lines, but it still
+gets the `## Content` marker. The content is everything after the first exact
+`## Content` line of that header, so a heading in the text such as
+`## Contents of the kit` stays part of the content. A file written before untitled
+entries got the marker is read whole.
+
+A cache file whose frontmatter is empty, isn't valid YAML, or isn't a mapping is
+treated as missing. Validation re-fetches the reference and rewrites the file,
+and tools that walk the cache skip the file with a warning.
+
 ### Cache Benefits
 
 - **Offline usage**: Work without network after initial fetch
@@ -254,6 +308,15 @@ The plugin discovers fields via:
 - `slot_uri` attribute (e.g., `slot_uri: dcterms:title`)
 - Fallback slot names (`reference`, `supporting_text`, `title`)
 
+Besides the URIs above, a URI of your own counts as a reference or excerpt
+field when its local name *ends* in the word. One `Field`, `Slot` or `Value`
+suffix may follow it, and for references also an identifier suffix such as
+`Id` or `Uri`. So `ex:reference`, `ex:cross_reference`, `ex:myReferenceField`
+and `ex:reference_id` are reference fields, and `ex:myExcerptField` is an
+excerpt field. `ex:referenceType`, `ex:excerpt_location` and `ex:excerpt_id`
+describe a reference or excerpt, so they are not. If a field of yours is missed
+(say `ex:excerpt_text`), declare it with `implements` or one of the URIs above.
+
 ## Editorial Conventions
 
 ### Square Brackets `[...]`

{linkml_reference_validator-0.3.1rc1 → linkml_reference_validator-0.5.0}/docs/how-to/fetch-full-text-and-pdfs.md

@@ -21,17 +21,34 @@ The validator tries providers in order until one returns usable full text.
 The default order is:
 
 ```
-pmc → epmc_preprint → unpaywall → openalex
+pmc → bookshelf → epmc_preprint → unpaywall → openalex
 ```
 
 - **`pmc`** — PubMed Central open-access subset (XML/HTML full text for many
   biomedical articles).
+- **`bookshelf`** — NCBI Bookshelf chapters, such as GeneReviews, read from a
+  local copy of NCBI's archive (see [NCBI Bookshelf](#ncbi-bookshelf-genereviews)
+  below). Does nothing until `bookshelf_archive_dir` is set.
 - **`epmc_preprint`** — Europe PMC preprint body text via the `fulltextRepo`
   PDF route (see [Preprints](#preprints) below).
 - **`unpaywall`** — Unpaywall open-access lookup by DOI (often a publisher or
   repository PDF).
 - **`openalex`** — OpenAlex open-access location (PDF/HTML).
 
+Two more built-in providers are available but not in the default chain; add
+them to `full_text_providers` to use them:
+
+- **`bioc`** — PMC Open Access articles as BioC passages, from NCBI's BioNLP
+  BioC API, by PMID. The reference list and the competing-interest,
+  author-contribution and funding statements are left out, so a snippet quoting
+  a cited paper cannot validate against the paper citing it.
+- **`epmc`** — Europe PMC's `fullTextXML` for open-access articles, by PMID.
+  This is the same JATS that `pmc` reads, served by Europe PMC rather than
+  NCBI, so after `pmc` it rarely finds anything new. It is useful as a second
+  route when NCBI is down or rate-limiting, or in place of `pmc` to avoid
+  depending on Entrez. Not to be confused with `epmc_preprint`, which handles
+  preprints only.
+
 Providers are skipped silently if they have nothing for a given reference, so
 the chain degrades gracefully: a miss in `pmc` simply moves on to `unpaywall`,
 and so on. If no provider yields full text, the reference keeps whatever
@@ -45,7 +62,8 @@ your config YAML or on the `ReferenceValidationConfig` object):
 | Key | Default | Description |
 |-----|---------|-------------|
 | `fetch_full_text` | `true` | Attempt to obtain full text via the provider chain when a metadata source does not already return full text. |
-| `full_text_providers` | `[pmc, epmc_preprint, unpaywall, openalex]` | Ordered list of provider names to try until one yields usable full text. |
+| `full_text_providers` | `[pmc, bookshelf, epmc_preprint, unpaywall, openalex]` | Ordered list of provider names to try until one yields usable full text. `bioc` and `epmc` are also built in but must be added explicitly. |
+| `bookshelf_archive_dir` | `null` | Directory of unpacked NCBI Bookshelf books, read by the `bookshelf` provider. |
 | `private_cache_dir` | `~/.cache/linkml-reference-validator/private` | Separate research cache for closed/user-library full text; never read by validation. |
 | `pdf_backend` | `pypdf` | Name of the PDF text-extraction backend. |
 | `download_pdfs` | `true` | If true, persist downloaded PDFs to the files cache directory. |
@@ -66,6 +84,7 @@ email: you@example.org
 fetch_full_text: true
 full_text_providers:
   - pmc
+  - bookshelf
   - epmc_preprint
   - unpaywall
   - openalex
@@ -73,6 +92,91 @@ pdf_backend: pypdf
 download_pdfs: true
 ```
 
+## NCBI Bookshelf (GeneReviews)
+
+A PMID can point to a chapter on NCBI Bookshelf rather than a journal article.
+Every [GeneReviews](https://www.ncbi.nlm.nih.gov/books/NBK1116/) entry is one
+of these. Such a chapter has no PMC copy, so without help the reference keeps
+only its PubMed summary, and a quote from the chapter body fails as
+abstract-only.
+
+The archive described below is for **caching** a chapter, not for validating
+against it. Validate quotes against the cached document. Once a chapter has
+been fetched, its full text is in the reference cache (`PMID_31536184.md`),
+and every later validation reads that file. There's no archive and no network
+call. So the large download is a one-time step on the machine that builds the
+cache, not something every run or every collaborator needs. See
+[Validate against the cache, not the archive](#validate-against-the-cache-not-the-archive).
+
+Bookshelf's web pages can't be fetched by a script: NCBI answers with a
+reCAPTCHA page. Instead, the `bookshelf` provider reads chapters from NCBI's
+[litarch archive](https://ftp.ncbi.nlm.nih.gov/pub/litarch/), which you
+download once per book:
+
+```bash
+linkml-reference-validator cache bookshelf-download NBK1116 --archive-dir bookshelf
+```
+
+GeneReviews (`NBK1116`) is a 640 MB download. Only the chapter XML is kept,
+about 200 MB. Then point the config at the same directory:
+
+```yaml
+bookshelf_archive_dir: bookshelf
+```
+
+The PubMed record names the chapter, so no extra lookup is needed:
+`PMID:31536184` is `gene/glutaric-a1`, read from
+`bookshelf/gene_NBK1116/glutaric-a1.nxml`. The cached entry records
+`full_text_provider: bookshelf`, and `full_text_url` holds the chapter's
+Bookshelf page.
+
+A chapter is used only when its own `<license>` is `open-access`. GeneReviews
+allows copying for noncommercial research with credit and no modification. A
+cache holds a verbatim copy, and its `full_text_url` links back to the
+chapter, so the text goes into the public cache like PMC full text does.
+
+To refresh a book, run the download again: it replaces the book's folder. Do
+this to pick up new or revised chapters.
+
+**References cached before the archive was set up** keep their abstract. The
+chain recorded that it couldn't read the chapter (`full_text_declined`, or
+`full_text_attempted` on a cache from an older version), and it doesn't retry
+on its own. Re-fetch them:
+
+```bash
+linkml-reference-validator cache reference PMID:31536184 --force
+```
+
+### Validate against the cache, not the archive
+
+The archive is only read when a chapter is **fetched**. That happens the first
+time a reference is seen, and on an explicit `--force`. Validation reads the
+cached document like any other cached reference. In practice:
+
+1. On one machine, download the book, set `bookshelf_archive_dir`, and cache
+   the chapters you cite (`cache reference PMID:...`, or a validation run).
+2. Commit the reference cache: the `PMID_*.md` files, which hold the chapter
+   text. Don't commit the archive.
+3. Everywhere else, including CI and collaborators, validate as usual. No
+   archive is needed, `bookshelf_archive_dir` can stay unset, and nothing is
+   downloaded from NCBI.
+
+What happens when the archive is missing:
+
+- **A cached chapter is never lost to a refresh that can't reach it.** If a
+  newer version of this tool marks the entry stale and re-fetches it, the
+  refresh gets only the abstract. The cached chapter is then kept and still
+  used, with a warning, and the entry stays stale until a run with the
+  archive refreshes it. For a committed cache you'd rather not re-fetch at
+  all, set `trust_cached_entries: true`.
+- **`--force` is the exception.** It replaces the entry with whatever the
+  fetch returns, so forcing a Bookshelf reference without the archive swaps
+  the chapter for its abstract (it warns as it does). Only force these
+  references where the archive is available.
+- **A reference that was never cached** still needs the archive once.
+  Without it, the reference is cached with its abstract only.
+  `full_text_declined: bookshelf_archive_not_configured` records why.
+
 ## Private manuscripts in Zotero
 
 The opt-in **`zotero`** provider can find non-open manuscripts that are already
@@ -270,7 +374,7 @@ the text came from. These provenance fields are written alongside the content:
 
 | Field | Description |
 |-------|-------------|
-| `full_text_provider` | Name of the provider that supplied the full text (e.g. `pmc`, `epmc_preprint`, `unpaywall`, or a custom name). |
+| `full_text_provider` | Name of the provider that supplied the full text (e.g. `pmc`, `bioc`, `epmc`, `epmc_preprint`, `unpaywall`, or a custom name). |
 | `oa_status` | Open-access status reported by the provider (e.g. `gold`, `green`, `bronze`). |
 | `license` | License of the open-access copy, when known. |
 | `local_pdf_path` | Path (relative to the cache directory) to the downloaded PDF, when one was persisted. |
@@ -301,14 +405,18 @@ fetcher does not crawl repository download links. If every provider completes
 without usable text or an error, the reference retains its original abstract
 and records `full_text_attempted: true`.
 
-New `full_text_html` cache entries carry `html_full_text_version: 1`. Older
-Markdown HTML full-text entries without this stamp are refreshed on their next
-validation read. This migration covers all providers because cached plain text
+New `full_text_html` cache entries carry `html_full_text_version: 2`. Entries
+with an older stamp, or none, are refreshed on their next validation read.
+Version 2 keeps all of an article region's text rather than only its `<p>`
+text, so lists, tables, headings and bare `<div>` text can be quoted. This migration covers all providers because cached plain text
 cannot establish which HTML body structure was present. It does not invalidate
 current PDF, XML, or abstract caches.
 
-If refreshing an old HTML entry fails, validation does **not** fall back to that
-entry: its text may be repository metadata rather than article evidence. The
+If refreshing an entry stamped `html_full_text_version: 1` or later fails,
+validation falls back to it with a warning, as for other content types: it is a
+checked article body, though it may lack list and table text. If refreshing an
+entry with no stamp fails, validation does **not** fall back to that entry: its
+text may be repository metadata rather than article evidence. The
 file remains untouched for inventory/export, and the next validation run retries
 recovery. Once the source is available, normal fetching replaces it with a
 verified article or the available metadata/abstract. Other content types retain

{linkml_reference_validator-0.3.1rc1 → linkml_reference_validator-0.5.0}/docs/how-to/repair-validation-errors.md

@@ -26,7 +26,25 @@ The distinct letters `Æ/æ` and `Œ/œ` remain distinct from `AE/ae` and `OE/oe
 Equating these letters would accept spelling changes beyond typographic repair.
 Validation uses an explicit ligature table instead of broad Unicode NFKC
 normalization so this change does not equate scientific notation such as `H₂O`
-with `H2O`, `10⁶` with `106`, or the micro sign `µ` with Greek `μ`.
+with `H2O` or `10⁶` with `106`. The micro sign `µ` and Greek `μ` *are* treated
+as the same letter, since sources use them interchangeably (`5 µM` matches
+`5 μM`), as are the variant Greek forms `ϕ ϵ ϑ ϐ ϖ ϰ ϱ` and the ordinary
+letters they stand for.
+
+### Unmapped PDF glyphs do not match
+
+Some PDFs draw a glyph without saying which character it is, so text extraction
+gets no character at all. This is common for `fi`/`fl` ligatures, and sometimes
+happens for digits or minus signs. PDF extraction writes U+FFFD (`�`) in its
+place and logs a warning with the count, so cached text reads `in�ammation`.
+Validation does not guess the missing character: a quote containing
+`inflammation` will not match that text. Check the cached file for `�` and
+quote around the damaged word or supply better full text.
+
+Matching treats `�` like punctuation, so the damaged word splits in two: the
+cached text above reads as `in ammation`. A quote that starts or ends inside
+the damaged word still matches, so `ammation of the gut` matches
+`in�ammation of the gut`.
 
 ## Quick Start
 
@@ -72,6 +90,18 @@ The repair command uses confidence thresholds to determine how to handle each er
 | **LOW** | 0.50-0.80 | Flag for manual review |
 | **VERY_LOW** | 0.00-0.50 | Recommend removal |
 
+The score ranges are the defaults. A fuzzy match is graded by the repair
+config's `auto_fix_threshold`, `suggest_threshold` and `removal_threshold`, so
+raising `auto_fix_threshold` (or passing `--auto-fix-threshold`) turns
+borderline auto-fixes into suggestions, and lowering it does the reverse. The
+three must be in order, `removal <= suggest <= auto_fix`. A lower threshold
+you leave unset follows one you set below it (`auto_fix_threshold: 0.75` alone
+also lowers `suggest_threshold` to 0.75), and `--auto-fix-threshold` lowers the
+other two with it. Thresholds you set out of order are an error.
+The flag overrides the config file only when it is given. Character
+normalization and ellipsis repairs are not scored, so the thresholds do not
+apply to them.
+
 ## Repair Strategies
 
 ### 1. Character Normalization (High Confidence)
@@ -90,11 +120,30 @@ supporting_text: "CO₂ levels were measured"
 
 Common mappings:
 - `CO2` → `CO₂`
+- `H2O2` → `H₂O₂`
 - `H2O` → `H₂O`
 - `O2` → `O₂`
 - `+/-` → `±`
 - `+-` → `±`
 
+Where a mapping starts or ends with a letter or digit, it applies only if no
+letter or digit sits directly next to it on that side. So `O2` is left alone in
+`CDKN2A` and `N2` in `HEK293N2`. The same rule also skips formulas joined to a
+coefficient or prefix, such as `6CO2`, `2H2O2` and `pCO2`. Fix those by hand.
+A mapping that starts and ends with punctuation, such as `+/-`, applies
+anywhere, so `5+/-2` becomes `5±2`. All mappings are applied in one pass,
+longest first.
+
+Validation reads `+-` (except before a letter), `+/-` and `±` all as the same
+word, ignores other punctuation, spells out Greek letters and folds ligatures,
+so it cannot confirm a mapping that changes only those, such as `+-` → `±` or a
+custom `alpha` → `α`. Such a mapping is kept only when the reference contains
+the repaired text exactly as written, apart from case and whitespace.
+That is what keeps `Ca2+-dependent` from becoming `Ca2±dependent`.
+
+Quotes with `...` or `[editorial brackets]` are checked part by part, the same
+way validation checks them.
+
 ### 2. Ellipsis Insertion (Medium Confidence)
 
 When text parts exist in the reference but aren't contiguous:
@@ -121,6 +170,22 @@ Suggested fix (85%): "Haemophilus influenzae type b" → "H. influenzae type b"
 
 Use with caution - verify the suggestion preserves the intended meaning.
 
+A correction that would change what the quote claims is reported, but never
+offered or applied as the fix:
+
+- **It drops a word of the quote,** such as "no" or "in mice".
+- **It swaps a word for one that differs only in a prefix,** such as
+  "unaffected" for "affected", "hyperphosphorylated" for
+  "hypophosphorylated", or "intracellular" for "extracellular". This holds
+  even when the quote also has a typo ("unaffectd").
+- **It adds a negation** the quote doesn't have, such as "not", "never",
+  "without" or "failed to".
+
+Words of three letters or fewer must match exactly, so "in" is never
+corrected to "on". These checks err towards caution, so some real typos go
+unfixed and need fixing by hand: a typo in a word's first few letters can look
+like a changed prefix ("ctivation" reads as "activation" without its "a").
+
 ### 4. Removal Recommendation (Very Low Confidence)
 
 Flags text that appears fabricated or hallucinated:
@@ -214,8 +279,8 @@ linkml-reference-validator repair data <DATA_FILE> --schema <SCHEMA> [OPTIONS]
 - `--schema PATH` - Path to LinkML schema (required)
 - `--target-class CLASS` - Target class to validate
 - `--dry-run / --no-dry-run` - Show changes without applying (default: dry-run)
-- `--auto-fix-threshold FLOAT` - Minimum similarity for auto-fixes (default: 0.95)
-- `--output PATH` - Output file path (default: overwrite with backup)
+- `--auto-fix-threshold FLOAT` - Minimum similarity for auto-fixes (default: the config file's `auto_fix_threshold`, else 0.95)
+- `--output PATH` - Output file path (default: overwrite the input, keeping a `.bak` backup). When given with `--no-dry-run`, it is always written, as an unchanged copy of the input when there is nothing to fix, and the input itself is not touched, so no backup is made
 - `--config PATH` - Path to repair configuration file
 - `--cache-dir PATH` - Directory for caching references
 - `--verbose` - Show detailed output

{linkml_reference_validator-0.3.1rc1 → linkml_reference_validator-0.5.0}/docs/how-to/use-local-files-and-urls.md

@@ -141,14 +141,95 @@ linkml-reference-validator validate text \
   `-webkit-text-security`, a `unicode-bidi` override, `<bdo>`, or a bidi
   control character in the text.
 
+**Edges.** A connector between two labelled nodes also gives an edge line,
+`"<source> <edge label> <target>"`, so a relational quote such as
+`"PTEN inhibits AKT"` can be checked. The connector is an element (usually a
+`<g>` holding the line and its label) whose `data-source` and `data-target`
+attributes name the two nodes' ids:
+
+```xml
+<defs><marker id="arrow"><path d="M0 0 L10 5 L0 10 z"/></marker></defs>
+<g id="pten"><rect x="0" y="0" width="60" height="30"/><text x="10" y="20">PTEN</text></g>
+<g id="akt"><rect x="200" y="0" width="60" height="30"/><text x="210" y="20">AKT</text></g>
+<g data-source="pten" data-target="akt">
+  <path d="M60 15 L200 15" stroke="black" marker-end="url(#arrow)"/>
+  <text x="110" y="10">inhibits</text>
+</g>
+```
+
+The attributes are not drawn, so they are only believed when the drawing
+agrees: the connector's stroked path must join the two nodes' shapes, one end
+on each (transforms applied, with a little slack for an arrowhead), and the
+connector, its label and both nodes and their labels must all be visible.
+
+Arrowheads drawn as markers give the direction:
+
+- **One arrowhead** (a `marker-start` or `marker-end`, set on the path or
+  inherited) points at the node at that end, and the line reads from the other
+  node to it. This holds even when it's the reverse of `data-source` →
+  `data-target`: the drawing wins, as in drawio.
+- **Arrowheads at both ends** make the line quotable both ways.
+- **No arrowhead** (a plain line, or a marker that draws nothing) reads in the
+  attributes' order, `data-source` first. This is a known limitation: nothing
+  drawn gives that order, so swapping the two attributes reverses the line
+  for the same picture.
+- **An arrowhead drawn as a separate shape** gives no edge line: which way
+  such a shape points can't be read, and the attributes' order could
+  contradict it. This is any other `<polygon>`, `<polyline>`, `<path>` or
+  `<line>` in the connector, wherever it sits; any `<use>` or `<image>`; and a
+  `<rect>`, `<circle>` or `<ellipse>` at an end of the line or on a node. A
+  `<rect>`, `<circle>` or `<ellipse>` in mid-line, such as a label's
+  background, is not taken for an arrowhead.
+- **Arrowheads at a bent line's inner vertices** (`marker-mid`, or the
+  `marker` shorthand) give no edge line, since each may point either way.
+
+A marker counts as an arrowhead when it names a `<marker>` in the figure that
+draws something: a shape that paints, at a size above zero. When that can't be
+told from the markup (a `var()` value, or a `<use>`, text or image inside the
+marker), the connector gives no edge line, since neither direction is safe. The
+marker's `orient` is not read: a marker drawn at a fixed angle is taken to
+point along the path. A node is the shapes it draws inside the
+element its id names; its text, and shapes that are hidden or paint nothing,
+do not count towards its box. Geometry the extractor cannot place from
+attributes gives no edge line either: CSS `transform` or other CSS geometry,
+`clip-path`, `mask` or `filter`, or an animation that moves shapes. A node drawn only through `<use>`, inside a
+nested `<svg>`, or sized in `%` or `em` gives no edge line.
+
 Known limits: a label drawn as several `<text>` elements, one per line, is
 read as several labels, so quote its lines with `...` between them. Text
 clipped by `clip-path` or `mask`, placed outside the
 `viewBox`, scaled to nothing, covered by another shape, drawn in the
 background colour, or cut short by CSS `text-overflow` or `-webkit-line-clamp`
 is still read. Text converted to outlines is not text and
-cannot be quoted. Native `.drawio` files and connector (edge) labels between
-nodes are not yet supported.
+cannot be quoted.
+
+## Diagrams as References (drawio)
+
+A `file:` reference can also point at a native drawio diagram, `.drawio` or
+`.drawio.xml`, compressed or not. It is read the same way as an SVG: one label
+per drawn cell, quotes held to one label, and an edge line
+`"<source> <edge label> <target>"` for each drawn edge between two labelled
+cells. In drawio an edge's source and target are what it is drawn between, so
+no geometry check is needed; its line and both cells' labels must still be
+visible. The arrows give the direction, as for SVG: an edge drawn pointing
+back at its source reads target first, one with arrows at both ends reads both
+ways, and one with no arrow reads source first. Only markers that point count
+as arrows: arrowheads (`classic`, `block`, `open`, `async` and their thin or
+double forms) and the inhibition T-bar `baseDash`. An oval, diamond, dash or
+ER symbol at an end is not an arrow. A marker drawio doesn't list as either
+gives no edge line. Every page of the file is read.
+
+A `.drawio.svg` file (drawio's editable SVG export) is read as an SVG: its
+labels are quotable, but it carries no `data-source`/`data-target`, so it
+gives no edge lines. Point at the `.drawio` source to quote its edges.
+
+Not read: cells marked invisible, in a hidden layer, or inside a collapsed
+group; labels switched off (`noLabel`) or drawn with zero `opacity`,
+`textOpacity` or `fontSize`, or with no `fontColor`; layer and page names,
+tooltips, styles and other attributes. HTML labels follow the same visibility
+rules as SVG `<foreignObject>` labels. A label computed from placeholders
+(`placeholders="1"`), or HTML that cannot be read statically, makes the
+reference `unavailable`.
 
 ## Using in Data Files
 
@@ -190,7 +271,8 @@ Both file and URL references work in LinkML data files:
 
 ## Limitations
 
-- **Local files are read as text**, apart from `.svg` figures (see above).
+- **Local files are read as text**, apart from `.svg`, `.drawio` and `.drawio.xml` figures
+  (see above).
   An `.html` file is matched against its raw markup.
 - **Local PDF files**: Not yet supported as `file:` references. PDFs fetched by `url:` are supported.
 - **Authentication**: URLs requiring login are not supported

{linkml_reference_validator-0.3.1rc1 → linkml_reference_validator-0.5.0}/docs/how-to/validate-titles.md

@@ -106,6 +106,23 @@ classes:
   title: "MUC1 oncoprotein blocks nuclear targeting of c-Abl"
 ```
 
+The reference may take the same shapes as for excerpts:
+
+- **A list of one** is checked like a single reference.
+- **Several references with one title field** pass if the title is any one
+  reference's title, since it names one paper. If it is none of them, a single
+  error names each reference with its own reason. A reference whose prefix is
+  skipped vouches for nothing.
+- **Reference objects with their own `title`**, listed or not, and with no title
+  field beside them, each have that title checked against their own reference.
+- **A reference with no usable identifier** (an object without `id`, an empty
+  string) is reported as an error, not skipped.
+
+With an excerpt as well, the title field is checked the same way: against the
+one reference, or, with several, passing if it is any one reference's title.
+It replaces any `title` a reference object carries. Without a title field,
+each reference object's own `title` is checked with the excerpt.
+
 ## Combined Validation
 
 When both title and excerpt fields are present, both are validated together:

{linkml_reference_validator-0.3.1rc1 → linkml_reference_validator-0.5.0}/docs/how-to/validate-urls.md

@@ -227,7 +227,22 @@ Ensure the cached content contains the text you're referencing.
 - Use `linkml-reference-validator cache reference url:https://... --force` to update cached content when pages change
 - Periodically review cached URLs to ensure they're still accessible
 
-### 4. Mix Reference Types
+### 4. Cite Pages With Fragments
+
+A fragment such as `#page=54` is never sent to the server, so it isn't part of
+the cache key. `url:https://example.com/report.pdf#page=54` and
+`...report.pdf#page=55` share one cache file, `url_https___example.com_report.pdf.md`,
+and the document is fetched and extracted once. Each reference still carries
+its own page-anchored URL, and quotes are matched against the whole document,
+not the cited page. Query strings (`?v=2`) do stay part of the key.
+
+Caches written before this change hold one file per fragment, such as
+`url_https___example.com_report.pdf#page_54.md`. These are still read, until
+the reference is next fetched (for example with `cache reference ... --force`);
+the fresh entry goes to the shared file, after which the per-fragment files can
+be deleted.
+
+### 5. Mix Reference Types
 
 URL references work alongside PMIDs and DOIs:
 

{linkml_reference_validator-0.3.1rc1 → linkml_reference_validator-0.5.0}/docs/reference/cache-format.md

@@ -117,7 +117,22 @@ assert header.model_dump()["database"] == "Orphanet"
 ```
 
 The checker derives the filename from the stored `reference_id` by replacing
-`:`, `/`, `?`, and `=` with `_` and appending `.md`. DOI filenames are compared
+`:`, `/`, `?`, and `=` with `_` and appending `.md`. A name that would exceed 200
+bytes of UTF-8 is cut on a character boundary and ends in `_` plus the first 16
+hex digits of the SHA-256 of the full `reference_id` (lowercased for a DOI), so
+a long `url:` reference fits common filesystems' 255-byte limit and two long IDs
+sharing a prefix still get different files. Shorter names are unchanged.
+
+Because the replacement is lossy, distinct references can sanitize to the same
+name, such as `url:https://ex.org/a?b=c` and `url:https://ex.org/a/b/c`. The
+first one written keeps that name. When the name is already held by a file whose
+`reference_id` is a different reference, the other one is written under the same
+name ending in `_` and the 16-hex-digit hash of its own ID instead, and the
+checker accepts that name for it. An entry is never served for a reference other
+than the one it records: a hand-written entry whose `reference_id` is a
+placeholder, or an alias spelling such as `pubmed:123` in `PMID_123.md`, is now
+treated as a miss and fetched afresh under the disambiguated name. DOI filenames
+are compared
 case-insensitively, matching the fetcher's existing behavior. Other filenames must
 match exactly. This uses the ID already stored in the file, without applying
 configured prefix aliases or requiring a currently registered reference source.