linkml-reference-validator 0.3.0rc3__tar.gz → 0.4.0__tar.gz

linkml_reference_validator-0.3.0rc3/README.md → linkml_reference_validator-0.4.0/PKG-INFO

@@ -1,3 +1,24 @@
+Metadata-Version: 2.5
+Name: linkml_reference_validator
+Version: 0.4.0
+Summary: Validation of supporting text from references and publications
+Author-email: Chris Mungall <cjmungall@lbl.gov>
+License-Expression: Apache-2.0
+License-File: LICENSE
+Requires-Python: <4.0,>=3.10
+Requires-Dist: beautifulsoup4>=4.9.0
+Requires-Dist: biopython>=1.80
+Requires-Dist: jsonpath-ng>=1.6.0
+Requires-Dist: linkml-runtime>=1.9.4
+Requires-Dist: lxml>=5.0.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pypdf>=4.0.0
+Requires-Dist: rapidfuzz>=3.14.3
+Requires-Dist: requests>=2.28.0
+Requires-Dist: ruamel-yaml>=0.18.0
+Requires-Dist: typer>=0.9.0
+Description-Content-Type: text/markdown
+
 # linkml-reference-validator
 
 [![Tests](https://img.shields.io/badge/tests-91%20passing-brightgreen)]()
@@ -37,6 +58,28 @@ linkml-validate -s schema.yaml data.yaml \
 
 ---
 
+## Agent skill
+
+Install the [linkml-reference-validator workflow skill](skills/linkml-reference-validator/SKILL.md) with the
+[skills CLI](https://skills.sh/) (requires Node.js). Preview available skills first:
+
+```bash
+npx skills add linkml/linkml-reference-validator --list
+npx skills add linkml/linkml-reference-validator --skill linkml-reference-validator
+```
+
+Installation defaults to the current project. Use `-a codex` or
+`-a claude-code` to select an agent; add `-g` for your user-wide skills directory:
+
+```bash
+npx skills add linkml/linkml-reference-validator --skill linkml-reference-validator -a codex -g
+```
+
+The skill helps agents configure reference QC, interpret failures and coverage,
+and set up repository hooks and CI using bundled reference guides. Install the
+runtime separately as described above; installing the skill does not activate
+hooks or configure CI, credentials, or data sources.
+
 ## Why Use This Tool?
 
 Scientific data often includes claims supported by quotes from publications. But how do you know the quotes are accurate?

linkml_reference_validator-0.3.0rc3/PKG-INFO → linkml_reference_validator-0.4.0/README.md

@@ -1,24 +1,3 @@
-Metadata-Version: 2.5
-Name: linkml_reference_validator
-Version: 0.3.0rc3
-Summary: Validation of supporting text from references and publications
-Author-email: Chris Mungall <cjmungall@lbl.gov>
-License-Expression: Apache-2.0
-License-File: LICENSE
-Requires-Python: <4.0,>=3.10
-Requires-Dist: beautifulsoup4>=4.9.0
-Requires-Dist: biopython>=1.80
-Requires-Dist: jsonpath-ng>=1.6.0
-Requires-Dist: linkml-runtime>=1.9.4
-Requires-Dist: lxml>=4.9.0
-Requires-Dist: pydantic>=2.0.0
-Requires-Dist: pypdf>=4.0.0
-Requires-Dist: rapidfuzz>=3.14.3
-Requires-Dist: requests>=2.28.0
-Requires-Dist: ruamel-yaml>=0.18.0
-Requires-Dist: typer>=0.9.0
-Description-Content-Type: text/markdown
-
 # linkml-reference-validator
 
 [![Tests](https://img.shields.io/badge/tests-91%20passing-brightgreen)]()
@@ -58,6 +37,28 @@ linkml-validate -s schema.yaml data.yaml \
 
 ---
 
+## Agent skill
+
+Install the [linkml-reference-validator workflow skill](skills/linkml-reference-validator/SKILL.md) with the
+[skills CLI](https://skills.sh/) (requires Node.js). Preview available skills first:
+
+```bash
+npx skills add linkml/linkml-reference-validator --list
+npx skills add linkml/linkml-reference-validator --skill linkml-reference-validator
+```
+
+Installation defaults to the current project. Use `-a codex` or
+`-a claude-code` to select an agent; add `-g` for your user-wide skills directory:
+
+```bash
+npx skills add linkml/linkml-reference-validator --skill linkml-reference-validator -a codex -g
+```
+
+The skill helps agents configure reference QC, interpret failures and coverage,
+and set up repository hooks and CI using bundled reference guides. Install the
+runtime separately as described above; installing the skill does not activate
+hooks or configure CI, credentials, or data sources.
+
 ## Why Use This Tool?
 
 Scientific data often includes claims supported by quotes from publications. But how do you know the quotes are accurate?

linkml_reference_validator-0.4.0/benchmarks/repeated_quotes.py

@@ -0,0 +1,49 @@
+"""Time repeated matching against a local UTF-8 document, without network access.
+
+Run the same script and input with each checkout's package on PYTHONPATH:
+    uv run python benchmarks/repeated_quotes.py /path/to/cached/publication.md
+
+Only successful matching is timed; this is a CPU microbenchmark, not end-to-end
+CLI validation. It includes each validator's first normalization of the body.
+"""
+
+import argparse
+import json
+import re
+from pathlib import Path
+from statistics import median
+from time import perf_counter
+
+from linkml_reference_validator.models import ReferenceContent, ReferenceValidationConfig
+from linkml_reference_validator.validation.supporting_text_validator import SupportingTextValidator
+
+
+def main() -> None:
+    """Report median time to match evenly spaced literal excerpts from a document."""
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("document", type=Path)
+    parser.add_argument("--quotes", type=int, default=100)
+    parser.add_argument("--rounds", type=int, default=3)
+    args = parser.parse_args()
+    content = args.document.read_text(encoding="utf-8")
+    # Each query is taken verbatim from the source; avoid editorial-bracket and
+    # ellipsis syntax, which would change what the matching API searches for.
+    candidates = [m.group() for m in re.finditer(r"[\w ,;:-]{30,100}", content)]
+    if args.quotes < 1 or args.rounds < 1 or not candidates:
+        parser.error("positive --quotes/--rounds and a document with prose are required")
+    quotes = [candidates[i * len(candidates) // args.quotes] for i in range(args.quotes)]
+    reference = ReferenceContent(reference_id="TEST:benchmark", content=content)
+    elapsed = []
+    for _ in range(args.rounds):
+        validator = SupportingTextValidator(ReferenceValidationConfig())
+        start = perf_counter()
+        for quote in quotes:
+            if not validator.find_text_in_reference(quote, reference).found:
+                raise SystemExit(f"Excerpt unexpectedly failed validation: {quote!r}")
+        elapsed.append(perf_counter() - start)
+    print(json.dumps({"characters": len(content), "quotes": len(quotes),
+                      "rounds_seconds": elapsed, "median_seconds": median(elapsed)}))
+
+
+if __name__ == "__main__":
+    main()

{linkml_reference_validator-0.3.0rc3 → linkml_reference_validator-0.4.0}/docs/concepts/content-types.md

@@ -11,6 +11,8 @@ When fetching references, the validator tracks what type of content was retrieve
 | `abstract_only` | PMID, DOI | Only the abstract was available |
 | `summary` | GEO, BioProject, BioSample, ClinicalTrials | Database summary/description field |
 | `local_file` | file: | Content read from local filesystem |
+| `local_svg` | file: (`.svg`) | Text an SVG figure renders, a blank line between labels; a quote must fall within one label |
+| `local_drawio` | file: (`.drawio`) | Labels a drawio diagram draws, read like `local_svg` |
 | `url` | url: | Content fetched from web URL |
 | `unavailable` | Any | No content could be retrieved |
 

{linkml_reference_validator-0.3.0rc3 → linkml_reference_validator-0.4.0}/docs/how-to/fetch-full-text-and-pdfs.md

@@ -21,17 +21,34 @@ The validator tries providers in order until one returns usable full text.
 The default order is:
 
 ```
-pmc → epmc_preprint → unpaywall → openalex
+pmc → bookshelf → epmc_preprint → unpaywall → openalex
 ```
 
 - **`pmc`** — PubMed Central open-access subset (XML/HTML full text for many
   biomedical articles).
+- **`bookshelf`** — NCBI Bookshelf chapters, such as GeneReviews, read from a
+  local copy of NCBI's archive (see [NCBI Bookshelf](#ncbi-bookshelf-genereviews)
+  below). Does nothing until `bookshelf_archive_dir` is set.
 - **`epmc_preprint`** — Europe PMC preprint body text via the `fulltextRepo`
   PDF route (see [Preprints](#preprints) below).
 - **`unpaywall`** — Unpaywall open-access lookup by DOI (often a publisher or
   repository PDF).
 - **`openalex`** — OpenAlex open-access location (PDF/HTML).
 
+Two more built-in providers are available but not in the default chain; add
+them to `full_text_providers` to use them:
+
+- **`bioc`** — PMC Open Access articles as BioC passages, from NCBI's BioNLP
+  BioC API, by PMID. The reference list and the competing-interest,
+  author-contribution and funding statements are left out, so a snippet quoting
+  a cited paper cannot validate against the paper citing it.
+- **`epmc`** — Europe PMC's `fullTextXML` for open-access articles, by PMID.
+  This is the same JATS that `pmc` reads, served by Europe PMC rather than
+  NCBI, so after `pmc` it rarely finds anything new. It is useful as a second
+  route when NCBI is down or rate-limiting, or in place of `pmc` to avoid
+  depending on Entrez. Not to be confused with `epmc_preprint`, which handles
+  preprints only.
+
 Providers are skipped silently if they have nothing for a given reference, so
 the chain degrades gracefully: a miss in `pmc` simply moves on to `unpaywall`,
 and so on. If no provider yields full text, the reference keeps whatever
@@ -45,7 +62,8 @@ your config YAML or on the `ReferenceValidationConfig` object):
 | Key | Default | Description |
 |-----|---------|-------------|
 | `fetch_full_text` | `true` | Attempt to obtain full text via the provider chain when a metadata source does not already return full text. |
-| `full_text_providers` | `[pmc, epmc_preprint, unpaywall, openalex]` | Ordered list of provider names to try until one yields usable full text. |
+| `full_text_providers` | `[pmc, bookshelf, epmc_preprint, unpaywall, openalex]` | Ordered list of provider names to try until one yields usable full text. `bioc` and `epmc` are also built in but must be added explicitly. |
+| `bookshelf_archive_dir` | `null` | Directory of unpacked NCBI Bookshelf books, read by the `bookshelf` provider. |
 | `private_cache_dir` | `~/.cache/linkml-reference-validator/private` | Separate research cache for closed/user-library full text; never read by validation. |
 | `pdf_backend` | `pypdf` | Name of the PDF text-extraction backend. |
 | `download_pdfs` | `true` | If true, persist downloaded PDFs to the files cache directory. |
@@ -66,6 +84,7 @@ email: you@example.org
 fetch_full_text: true
 full_text_providers:
   - pmc
+  - bookshelf
   - epmc_preprint
   - unpaywall
   - openalex
@@ -73,6 +92,91 @@ pdf_backend: pypdf
 download_pdfs: true
 ```
 
+## NCBI Bookshelf (GeneReviews)
+
+A PMID can point to a chapter on NCBI Bookshelf rather than a journal article.
+Every [GeneReviews](https://www.ncbi.nlm.nih.gov/books/NBK1116/) entry is one
+of these. Such a chapter has no PMC copy, so without help the reference keeps
+only its PubMed summary, and a quote from the chapter body fails as
+abstract-only.
+
+The archive described below is for **caching** a chapter, not for validating
+against it. Validate quotes against the cached document. Once a chapter has
+been fetched, its full text is in the reference cache (`PMID_31536184.md`),
+and every later validation reads that file. There's no archive and no network
+call. So the large download is a one-time step on the machine that builds the
+cache, not something every run or every collaborator needs. See
+[Validate against the cache, not the archive](#validate-against-the-cache-not-the-archive).
+
+Bookshelf's web pages can't be fetched by a script: NCBI answers with a
+reCAPTCHA page. Instead, the `bookshelf` provider reads chapters from NCBI's
+[litarch archive](https://ftp.ncbi.nlm.nih.gov/pub/litarch/), which you
+download once per book:
+
+```bash
+linkml-reference-validator cache bookshelf-download NBK1116 --archive-dir bookshelf
+```
+
+GeneReviews (`NBK1116`) is a 640 MB download. Only the chapter XML is kept,
+about 200 MB. Then point the config at the same directory:
+
+```yaml
+bookshelf_archive_dir: bookshelf
+```
+
+The PubMed record names the chapter, so no extra lookup is needed:
+`PMID:31536184` is `gene/glutaric-a1`, read from
+`bookshelf/gene_NBK1116/glutaric-a1.nxml`. The cached entry records
+`full_text_provider: bookshelf`, and `full_text_url` holds the chapter's
+Bookshelf page.
+
+A chapter is used only when its own `<license>` is `open-access`. GeneReviews
+allows copying for noncommercial research with credit and no modification. A
+cache holds a verbatim copy, and its `full_text_url` links back to the
+chapter, so the text goes into the public cache like PMC full text does.
+
+To refresh a book, run the download again: it replaces the book's folder. Do
+this to pick up new or revised chapters.
+
+**References cached before the archive was set up** keep their abstract. The
+chain recorded that it couldn't read the chapter (`full_text_declined`, or
+`full_text_attempted` on a cache from an older version), and it doesn't retry
+on its own. Re-fetch them:
+
+```bash
+linkml-reference-validator cache reference PMID:31536184 --force
+```
+
+### Validate against the cache, not the archive
+
+The archive is only read when a chapter is **fetched**. That happens the first
+time a reference is seen, and on an explicit `--force`. Validation reads the
+cached document like any other cached reference. In practice:
+
+1. On one machine, download the book, set `bookshelf_archive_dir`, and cache
+   the chapters you cite (`cache reference PMID:...`, or a validation run).
+2. Commit the reference cache: the `PMID_*.md` files, which hold the chapter
+   text. Don't commit the archive.
+3. Everywhere else, including CI and collaborators, validate as usual. No
+   archive is needed, `bookshelf_archive_dir` can stay unset, and nothing is
+   downloaded from NCBI.
+
+What happens when the archive is missing:
+
+- **A cached chapter is never lost to a refresh that can't reach it.** If a
+  newer version of this tool marks the entry stale and re-fetches it, the
+  refresh gets only the abstract. The cached chapter is then kept and still
+  used, with a warning, and the entry stays stale until a run with the
+  archive refreshes it. For a committed cache you'd rather not re-fetch at
+  all, set `trust_cached_entries: true`.
+- **`--force` is the exception.** It replaces the entry with whatever the
+  fetch returns, so forcing a Bookshelf reference without the archive swaps
+  the chapter for its abstract (it warns as it does). Only force these
+  references where the archive is available.
+- **A reference that was never cached** still needs the archive once.
+  Without it, the reference is cached with its abstract only.
+  `full_text_declined: bookshelf_archive_not_configured` records why.
+
 ## Private manuscripts in Zotero
 
 The opt-in **`zotero`** provider can find non-open manuscripts that are already
@@ -270,7 +374,7 @@ the text came from. These provenance fields are written alongside the content:
 
 | Field | Description |
 |-------|-------------|
-| `full_text_provider` | Name of the provider that supplied the full text (e.g. `pmc`, `epmc_preprint`, `unpaywall`, or a custom name). |
+| `full_text_provider` | Name of the provider that supplied the full text (e.g. `pmc`, `bioc`, `epmc`, `epmc_preprint`, `unpaywall`, or a custom name). |
 | `oa_status` | Open-access status reported by the provider (e.g. `gold`, `green`, `bronze`). |
 | `license` | License of the open-access copy, when known. |
 | `local_pdf_path` | Path (relative to the cache directory) to the downloaded PDF, when one was persisted. |

{linkml_reference_validator-0.3.0rc3 → linkml_reference_validator-0.4.0}/docs/how-to/repair-validation-errors.md

@@ -28,6 +28,21 @@ Validation uses an explicit ligature table instead of broad Unicode NFKC
 normalization so this change does not equate scientific notation such as `H₂O`
 with `H2O`, `10⁶` with `106`, or the micro sign `µ` with Greek `μ`.
 
+### Unmapped PDF glyphs do not match
+
+Some PDFs draw a glyph without saying which character it is, so text extraction
+gets no character at all. This is common for `fi`/`fl` ligatures, and sometimes
+happens for digits or minus signs. PDF extraction writes U+FFFD (`�`) in its
+place and logs a warning with the count, so cached text reads `in�ammation`.
+Validation does not guess the missing character: a quote containing
+`inflammation` will not match that text. Check the cached file for `�` and
+quote around the damaged word or supply better full text.
+
+Matching treats `�` like punctuation, so the damaged word splits in two: the
+cached text above reads as `in ammation`. A quote that starts or ends inside
+the damaged word still matches, so `ammation of the gut` matches
+`in�ammation of the gut`.
+
 ## Quick Start
 
 ### Repair a Single Quote
@@ -121,6 +136,22 @@ Suggested fix (85%): "Haemophilus influenzae type b" → "H. influenzae type b"
 
 Use with caution - verify the suggestion preserves the intended meaning.
 
+A correction that would change what the quote claims is reported, but never
+offered or applied as the fix:
+
+- **It drops a word of the quote,** such as "no" or "in mice".
+- **It swaps a word for one that differs only in a prefix,** such as
+  "unaffected" for "affected", "hyperphosphorylated" for
+  "hypophosphorylated", or "intracellular" for "extracellular". This holds
+  even when the quote also has a typo ("unaffectd").
+- **It adds a negation** the quote doesn't have, such as "not", "never",
+  "without" or "failed to".
+
+Words of three letters or fewer must match exactly, so "in" is never
+corrected to "on". These checks err towards caution, so some real typos go
+unfixed and need fixing by hand: a typo in a word's first few letters can look
+like a changed prefix ("ctivation" reads as "activation" without its "a").
+
 ### 4. Removal Recommendation (Very Low Confidence)
 
 Flags text that appears fabricated or hallucinated:

linkml_reference_validator-0.4.0/docs/how-to/use-local-files-and-urls.md

@@ -0,0 +1,279 @@
+# Using Local Files and URLs as References
+
+In addition to PubMed IDs and DOIs, the validator supports local files and web URLs as reference sources. This is useful when your supporting text comes from internal documentation, research notes, or web pages.
+
+## Local File References
+
+Use the `file:` prefix to reference local files:
+
+```bash
+linkml-reference-validator validate text \
+  "JAK1 binds to the receptor complex" \
+  file:./research/jak-signaling-notes.md
+```
+
+### Supported File Types
+
+- **Markdown** (`.md`) - Title extracted from first `# heading`
+- **Plain text** (`.txt`) - Content used as-is
+- **HTML** (`.html`) - Content preserved including HTML entities
+
+### Path Resolution
+
+**Absolute paths** always work:
+
+```bash
+file:/Users/me/research/notes.md
+```
+
+**Relative paths** are resolved in order:
+
+1. If `reference_base_dir` is configured, paths resolve relative to it
+2. Otherwise, paths resolve relative to the current working directory
+
+Note: cached references use the resolved absolute path in the reference ID, so
+the cache file name reflects the full path.
+
+### Configuring a Base Directory
+
+Set a base directory for all relative file references:
+
+```python
+from linkml_reference_validator.models import ReferenceValidationConfig
+from pathlib import Path
+
+config = ReferenceValidationConfig(
+    reference_base_dir=Path("./references"),
+)
+```
+
+Then `file:notes.md` resolves to `./references/notes.md`.
+
+### Example: Validating Against Research Notes
+
+Create a research file:
+
+```markdown
+# JAK-STAT Signaling Pathway
+
+JAK1 binds to the receptor complex and initiates downstream signaling.
+This leads to STAT phosphorylation and nuclear translocation.
+```
+
+Validate:
+
+```bash
+linkml-reference-validator validate text \
+  "JAK1 binds to the receptor complex" \
+  file:./jak-signaling.md
+```
+
+## URL References
+
+Use the `url:` prefix to reference web pages (the prefix is optional for bare
+HTTP/HTTPS URLs and will be normalized to `url:` internally):
+
+```bash
+linkml-reference-validator validate text \
+  "Climate change affects biodiversity" \
+  url:https://example.org/climate-report.html
+```
+
+### Caching
+
+URLs are cached the same way as PMID and DOI references:
+
+- First fetch downloads and caches the content
+- Subsequent validations use the cached version
+- Use `linkml-reference-validator cache reference url:https://... --force` to re-fetch
+
+### Title Extraction
+
+For HTML pages, the title is the `citation_title` meta tag when present, then the `<title>` tag. For PDFs, the validator tries a publisher landing page's `citation_title` and then the PDF's embedded title. When neither is found, and for other content types, the URL itself is used as the title. See [Validating URLs](validate-urls.md#titles).
+
+### Example: Validating Against a Web Page
+
+```bash
+# First validation fetches and caches
+linkml-reference-validator validate text \
+  "The quick brown fox jumps over the lazy dog" \
+  url:https://example.com/pangram-examples.html
+
+# Subsequent validations use cache
+linkml-reference-validator validate text \
+  "A quick brown fox" \
+  url:https://example.com/pangram-examples.html
+```
+
+## Diagrams as References (SVG)
+
+A `file:` reference can point at an `.svg` figure, such as a pathway diagram.
+The quote is then matched against the text the figure **renders**, not its
+markup:
+
+```bash
+linkml-reference-validator validate text \
+  "loss of PTEN activates AKT" \
+  file:./figures/pathway.svg
+```
+
+- Each `<text>` element is one label. A label split over several `<tspan>`
+  lines can be quoted across the break.
+- A quote must fall within one label. A figure with a `PTEN` label and a
+  separate `activates AKT` label does not support `"PTEN activates AKT"`,
+  however close together they sit: neither label says it. To cite both, quote
+  them as separate parts, `"PTEN ... activates AKT"`.
+- HTML labels inside `<foreignObject>`, which is how drawio's **Export as SVG**
+  writes them, are read too.
+- Text the reader cannot see is never quotable. That covers `<desc>`, `<title>`,
+  `<metadata>`, comments, attribute values (`id`, `fill`, `class`, ...),
+  `<defs>`/`<symbol>` content that no `<use>` draws, and text hidden by
+  `display:none`, `visibility:hidden`, zero `opacity`, `fill="none"` or a zero
+  `font-size`, whether set on the element, inherited from a group, or set by a
+  `<style>` rule.
+- If the extractor cannot tell what the figure renders, the reference is
+  `unavailable` rather than read as raw markup. That happens when a `<style>`
+  rule hides or shows text through a selector other than a type, class or id
+  selector (`g > text`, `:first-child`, `@media`), when visibility depends on
+  `var()` or `calc()`, when a stylesheet is imported or linked, when the SVG
+  contains a `<script>` or animates visibility, or when something changes the
+  characters drawn: CSS `content`, a string `list-style-type` or `quotes`,
+  `-webkit-text-security`, a `unicode-bidi` override, `<bdo>`, or a bidi
+  control character in the text.
+
+**Edges.** A connector between two labelled nodes also gives an edge line,
+`"<source> <edge label> <target>"`, so a relational quote such as
+`"PTEN inhibits AKT"` can be checked. The connector is an element (usually a
+`<g>` holding the line and its label) whose `data-source` and `data-target`
+attributes name the two nodes' ids:
+
+```xml
+<defs><marker id="arrow"><path d="M0 0 L10 5 L0 10 z"/></marker></defs>
+<g id="pten"><rect x="0" y="0" width="60" height="30"/><text x="10" y="20">PTEN</text></g>
+<g id="akt"><rect x="200" y="0" width="60" height="30"/><text x="210" y="20">AKT</text></g>
+<g data-source="pten" data-target="akt">
+  <path d="M60 15 L200 15" stroke="black" marker-end="url(#arrow)"/>
+  <text x="110" y="10">inhibits</text>
+</g>
+```
+
+The attributes are not drawn, so they are only believed when the drawing
+agrees: the connector's stroked path must join the two nodes' shapes, one end
+on each (transforms applied, with a little slack for an arrowhead), and the
+connector, its label and both nodes and their labels must all be visible.
+
+Arrowheads drawn as markers give the direction:
+
+- **One arrowhead** (a `marker-start` or `marker-end`, set on the path or
+  inherited) points at the node at that end, and the line reads from the other
+  node to it. This holds even when it's the reverse of `data-source` →
+  `data-target`: the drawing wins, as in drawio.
+- **Arrowheads at both ends** make the line quotable both ways.
+- **No arrowhead** (a plain line, or a marker that draws nothing) reads in the
+  attributes' order, `data-source` first. This is a known limitation: nothing
+  drawn gives that order, so swapping the two attributes reverses the line
+  for the same picture.
+- **An arrowhead drawn as a separate shape** gives no edge line: which way
+  such a shape points can't be read, and the attributes' order could
+  contradict it. This is any other `<polygon>`, `<polyline>`, `<path>` or
+  `<line>` in the connector, wherever it sits; any `<use>` or `<image>`; and a
+  `<rect>`, `<circle>` or `<ellipse>` at an end of the line or on a node. A
+  `<rect>`, `<circle>` or `<ellipse>` in mid-line, such as a label's
+  background, is not taken for an arrowhead.
+- **Arrowheads at a bent line's inner vertices** (`marker-mid`, or the
+  `marker` shorthand) give no edge line, since each may point either way.
+
+A marker counts as an arrowhead when it names a `<marker>` in the figure that
+draws something: a shape that paints, at a size above zero. When that can't be
+told from the markup (a `var()` value, or a `<use>`, text or image inside the
+marker), the connector gives no edge line, since neither direction is safe. The
+marker's `orient` is not read: a marker drawn at a fixed angle is taken to
+point along the path. A node is the shapes it draws inside the
+element its id names; its text, and shapes that are hidden or paint nothing,
+do not count towards its box. Geometry the extractor cannot place from
+attributes gives no edge line either: CSS `transform` or other CSS geometry,
+`clip-path`, `mask` or `filter`, or an animation that moves shapes. A node drawn only through `<use>`, inside a
+nested `<svg>`, or sized in `%` or `em` gives no edge line.
+
+Known limits: a label drawn as several `<text>` elements, one per line, is
+read as several labels, so quote its lines with `...` between them. Text
+clipped by `clip-path` or `mask`, placed outside the
+`viewBox`, scaled to nothing, covered by another shape, drawn in the
+background colour, or cut short by CSS `text-overflow` or `-webkit-line-clamp`
+is still read. Text converted to outlines is not text and
+cannot be quoted.
+
+## Diagrams as References (drawio)
+
+A `file:` reference can also point at a native drawio diagram, `.drawio` or
+`.drawio.xml`, compressed or not. It is read the same way as an SVG: one label
+per drawn cell, quotes held to one label, and an edge line
+`"<source> <edge label> <target>"` for each drawn edge between two labelled
+cells. In drawio an edge's source and target are what it is drawn between, so
+no geometry check is needed; its line and both cells' labels must still be
+visible. The arrows give the direction, as for SVG: an edge drawn pointing
+back at its source reads target first, one with arrows at both ends reads both
+ways, and one with no arrow reads source first. Only markers that point count
+as arrows: arrowheads (`classic`, `block`, `open`, `async` and their thin or
+double forms) and the inhibition T-bar `baseDash`. An oval, diamond, dash or
+ER symbol at an end is not an arrow. A marker drawio doesn't list as either
+gives no edge line. Every page of the file is read.
+
+A `.drawio.svg` file (drawio's editable SVG export) is read as an SVG: its
+labels are quotable, but it carries no `data-source`/`data-target`, so it
+gives no edge lines. Point at the `.drawio` source to quote its edges.
+
+Not read: cells marked invisible, in a hidden layer, or inside a collapsed
+group; labels switched off (`noLabel`) or drawn with zero `opacity`,
+`textOpacity` or `fontSize`, or with no `fontColor`; layer and page names,
+tooltips, styles and other attributes. HTML labels follow the same visibility
+rules as SVG `<foreignObject>` labels. A label computed from placeholders
+(`placeholders="1"`), or HTML that cannot be read statically, makes the
+reference `unavailable`.
+
+## Using in Data Files
+
+Both file and URL references work in LinkML data files:
+
+```yaml
+# data.yaml
+- id: local-evidence
+  supporting_text: JAK1 binds to the receptor complex
+  reference: file:./research/jak-notes.md
+
+- id: web-evidence
+  supporting_text: Climate impacts are accelerating
+  reference: url:https://example.org/climate-report.html
+```
+
+## Reference Type Summary
+
+| Prefix | Example | Source |
+|--------|---------|--------|
+| `PMID:` | `PMID:16888623` | PubMed via NCBI Entrez |
+| `DOI:` | `DOI:10.1038/nature12373` | Crossref API |
+| `file:` | `file:./notes.md` | Local filesystem |
+| `url:` | `url:https://example.com` | Web (HTTP/HTTPS) |
+
+## Best Practices
+
+### For Local Files
+
+- Keep reference files in a dedicated directory
+- Use `reference_base_dir` for consistent path resolution
+- Use markdown for structured content with clear headings
+
+### For URLs
+
+- Prefer stable URLs (avoid query parameters that change)
+- Be aware that web content may change (cache helps with reproducibility)
+- Consider downloading important pages as local files for long-term stability
+
+## Limitations
+
+- **Local files are read as text**, apart from `.svg`, `.drawio` and `.drawio.xml` figures
+  (see above).
+  An `.html` file is matched against its raw markup.
+- **Local PDF files**: Not yet supported as `file:` references. PDFs fetched by `url:` are supported.
+- **Authentication**: URLs requiring login are not supported
+- **Dynamic content**: JavaScript-rendered pages may not work