linkml-reference-validator 0.4.0__tar.gz → 0.5.1__tar.gz

{linkml_reference_validator-0.4.0 → linkml_reference_validator-0.5.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: linkml_reference_validator
-Version: 0.4.0
+Version: 0.5.1
 Summary: Validation of supporting text from references and publications
 Author-email: Chris Mungall <cjmungall@lbl.gov>
 License-Expression: Apache-2.0

{linkml_reference_validator-0.4.0 → linkml_reference_validator-0.5.1}/docs/concepts/how-it-works.md

@@ -13,7 +13,23 @@ linkml-reference-validator validates that quoted text (supporting text) actually
 Before matching, both the supporting text and reference content are normalized:
 
 - **Lowercased**: `"MUC1"` → `"muc1"`
-- **Punctuation removed**: `"c-Abl"` → `"c abl"`
+- **Punctuation removed**: `"c-Abl"` → `"c abl"` (a point or comma between
+  digits is kept, so `"10.5"` stays one number)
+- **Meaning-bearing symbols kept as words**: `<` `>` `≤` `≥` become `lt` `gt`
+  `le` `ge`, `±` (or `+/-`) becomes `pm`, and a minus sign before a number stays
+  on it, so `"p < 0.05"` does not match `"p > 0.05"` and `"2.5 fold"` does not
+  match `"-2.5 fold"`. ASCII and Unicode spellings are equivalent (`<=` and `≤`,
+  `-` and `−`). Publisher spellings count too: MEDLINE's `>/=` and `</=` for
+  `≥` and `≤`, a middle-dot decimal (`57·4` is `57.4`), and an en dash or
+  Unicode hyphen printed as a minus sign (`z score, –1.98`) or inside `+/–`.
+  A `−` set off from its number by a space (`from − 2.92`) is still a sign,
+  unless a number comes before it (`10 − 2` is a subtraction). A hyphen or en
+  dash after a word or digit (`IL-6`, `3-5`, `1–12`) is still punctuation. A
+  dash after a space and before a digit is a sign, so a range spaced on one
+  side only (`18 –65`) reads as `18` and `-65`; quote it as the source
+  prints it. A middle dot after a letter-joined number is not a decimal
+  point, so `MgCl2·6H2O` keeps its hydrate dot (and `pH7·4` does not read as
+  `pH7.4`; `pH 7·4` does).
 - **Whitespace collapsed**: Multiple spaces become single space
 - **Editorial notes removed**: `"[mucin 1]"` → `""`
 
@@ -36,6 +52,34 @@ reference_content = "...The MUC1 oncoprotein blocks nuclear..."
 # Match: "muc1 oncoprotein" found in normalized reference
 ```
 
+**Matches start and end on whole words:**
+
+A quote may not begin or end inside a word of the reference. Otherwise a quote
+stating the opposite of the paper would pass:
+
+```python
+supporting_text = "activates AKT signaling"
+reference_content = "...Loss of PTEN inactivates AKT signaling..."
+# NOT FOUND: "activates" only occurs inside "inactivates"
+
+supporting_text = "5 mM H2O2"
+reference_content = "...treated with 10.5 mM H2O2..."
+# NOT FOUND: "5" only occurs inside "10.5"
+```
+
+Each part of a quote split with `...` is held to the same rule.
+
+One allowance: when a quote ends in a letter, the reference word it ends on may
+carry trailing digits. Full-text extraction welds superscript citations onto the
+word before them, so `tumor suppressor<sup>12</sup>` reads as `suppressor12`,
+and a quote of `PTEN is a tumor suppressor` must still match. A quote ending in
+a digit gets no allowance (`10` never matches `105`). The same allowance means a
+quote ending in `AKT` matches `AKT1`, as plain substring matching always did.
+
+Because a point or comma between digits stays part of the number, numbers must
+be quoted with the reference's own separators: `days 3, 5 and 7` does not match
+`days 3,5 and 7`, and `10 000` does not match `10,000`.
+
 **Empty excerpts are rejected:**
 
 An empty or whitespace-only excerpt is always an error, never a pass. The
@@ -224,6 +268,16 @@ content_type: abstract_only
 The MUC1 oncoprotein blocks nuclear targeting...
 ```
 
+An entry with no title has no `# title`, authors or journal lines, but it still
+gets the `## Content` marker. The content is everything after the first exact
+`## Content` line of that header, so a heading in the text such as
+`## Contents of the kit` stays part of the content. A file written before untitled
+entries got the marker is read whole.
+
+A cache file whose frontmatter is empty, isn't valid YAML, or isn't a mapping is
+treated as missing. Validation re-fetches the reference and rewrites the file,
+and tools that walk the cache skip the file with a warning.
+
 ### Cache Benefits
 
 - **Offline usage**: Work without network after initial fetch
@@ -254,6 +308,15 @@ The plugin discovers fields via:
 - `slot_uri` attribute (e.g., `slot_uri: dcterms:title`)
 - Fallback slot names (`reference`, `supporting_text`, `title`)
 
+Besides the URIs above, a URI of your own counts as a reference or excerpt
+field when its local name *ends* in the word. One `Field`, `Slot` or `Value`
+suffix may follow it, and for references also an identifier suffix such as
+`Id` or `Uri`. So `ex:reference`, `ex:cross_reference`, `ex:myReferenceField`
+and `ex:reference_id` are reference fields, and `ex:myExcerptField` is an
+excerpt field. `ex:referenceType`, `ex:excerpt_location` and `ex:excerpt_id`
+describe a reference or excerpt, so they are not. If a field of yours is missed
+(say `ex:excerpt_text`), declare it with `implements` or one of the URIs above.
+
 ## Editorial Conventions
 
 ### Square Brackets `[...]`

{linkml_reference_validator-0.4.0 → linkml_reference_validator-0.5.1}/docs/how-to/fetch-full-text-and-pdfs.md

@@ -405,14 +405,18 @@ fetcher does not crawl repository download links. If every provider completes
 without usable text or an error, the reference retains its original abstract
 and records `full_text_attempted: true`.
 
-New `full_text_html` cache entries carry `html_full_text_version: 1`. Older
-Markdown HTML full-text entries without this stamp are refreshed on their next
-validation read. This migration covers all providers because cached plain text
+New `full_text_html` cache entries carry `html_full_text_version: 2`. Entries
+with an older stamp, or none, are refreshed on their next validation read.
+Version 2 keeps all of an article region's text rather than only its `<p>`
+text, so lists, tables, headings and bare `<div>` text can be quoted. This migration covers all providers because cached plain text
 cannot establish which HTML body structure was present. It does not invalidate
 current PDF, XML, or abstract caches.
 
-If refreshing an old HTML entry fails, validation does **not** fall back to that
-entry: its text may be repository metadata rather than article evidence. The
+If refreshing an entry stamped `html_full_text_version: 1` or later fails,
+validation falls back to it with a warning, as for other content types: it is a
+checked article body, though it may lack list and table text. If refreshing an
+entry with no stamp fails, validation does **not** fall back to that entry: its
+text may be repository metadata rather than article evidence. The
 file remains untouched for inventory/export, and the next validation run retries
 recovery. Once the source is available, normal fetching replaces it with a
 verified article or the available metadata/abstract. Other content types retain

{linkml_reference_validator-0.4.0 → linkml_reference_validator-0.5.1}/docs/how-to/repair-validation-errors.md

@@ -26,7 +26,10 @@ The distinct letters `Æ/æ` and `Œ/œ` remain distinct from `AE/ae` and `OE/oe
 Equating these letters would accept spelling changes beyond typographic repair.
 Validation uses an explicit ligature table instead of broad Unicode NFKC
 normalization so this change does not equate scientific notation such as `H₂O`
-with `H2O`, `10⁶` with `106`, or the micro sign `µ` with Greek `μ`.
+with `H2O` or `10⁶` with `106`. The micro sign `µ` and Greek `μ` *are* treated
+as the same letter, since sources use them interchangeably (`5 µM` matches
+`5 μM`), as are the variant Greek forms `ϕ ϵ ϑ ϐ ϖ ϰ ϱ` and the ordinary
+letters they stand for.
 
 ### Unmapped PDF glyphs do not match
 
@@ -87,6 +90,18 @@ The repair command uses confidence thresholds to determine how to handle each er
 | **LOW** | 0.50-0.80 | Flag for manual review |
 | **VERY_LOW** | 0.00-0.50 | Recommend removal |
 
+The score ranges are the defaults. A fuzzy match is graded by the repair
+config's `auto_fix_threshold`, `suggest_threshold` and `removal_threshold`, so
+raising `auto_fix_threshold` (or passing `--auto-fix-threshold`) turns
+borderline auto-fixes into suggestions, and lowering it does the reverse. The
+three must be in order, `removal <= suggest <= auto_fix`. A lower threshold
+you leave unset follows one you set below it (`auto_fix_threshold: 0.75` alone
+also lowers `suggest_threshold` to 0.75), and `--auto-fix-threshold` lowers the
+other two with it. Thresholds you set out of order are an error.
+The flag overrides the config file only when it is given. Character
+normalization and ellipsis repairs are not scored, so the thresholds do not
+apply to them.
+
 ## Repair Strategies
 
 ### 1. Character Normalization (High Confidence)
@@ -105,11 +120,30 @@ supporting_text: "CO₂ levels were measured"
 
 Common mappings:
 - `CO2` → `CO₂`
+- `H2O2` → `H₂O₂`
 - `H2O` → `H₂O`
 - `O2` → `O₂`
 - `+/-` → `±`
 - `+-` → `±`
 
+Where a mapping starts or ends with a letter or digit, it applies only if no
+letter or digit sits directly next to it on that side. So `O2` is left alone in
+`CDKN2A` and `N2` in `HEK293N2`. The same rule also skips formulas joined to a
+coefficient or prefix, such as `6CO2`, `2H2O2` and `pCO2`. Fix those by hand.
+A mapping that starts and ends with punctuation, such as `+/-`, applies
+anywhere, so `5+/-2` becomes `5±2`. All mappings are applied in one pass,
+longest first.
+
+Validation reads `+-` (except before a letter), `+/-` and `±` all as the same
+word, ignores other punctuation, spells out Greek letters and folds ligatures,
+so it cannot confirm a mapping that changes only those, such as `+-` → `±` or a
+custom `alpha` → `α`. Such a mapping is kept only when the reference contains
+the repaired text exactly as written, apart from case and whitespace.
+That is what keeps `Ca2+-dependent` from becoming `Ca2±dependent`.
+
+Quotes with `...` or `[editorial brackets]` are checked part by part, the same
+way validation checks them.
+
 ### 2. Ellipsis Insertion (Medium Confidence)
 
 When text parts exist in the reference but aren't contiguous:
@@ -245,8 +279,8 @@ linkml-reference-validator repair data <DATA_FILE> --schema <SCHEMA> [OPTIONS]
 - `--schema PATH` - Path to LinkML schema (required)
 - `--target-class CLASS` - Target class to validate
 - `--dry-run / --no-dry-run` - Show changes without applying (default: dry-run)
-- `--auto-fix-threshold FLOAT` - Minimum similarity for auto-fixes (default: 0.95)
-- `--output PATH` - Output file path (default: overwrite with backup)
+- `--auto-fix-threshold FLOAT` - Minimum similarity for auto-fixes (default: the config file's `auto_fix_threshold`, else 0.95)
+- `--output PATH` - Output file path (default: overwrite the input, keeping a `.bak` backup). When given with `--no-dry-run`, it is always written, as an unchanged copy of the input when there is nothing to fix, and the input itself is not touched, so no backup is made
 - `--config PATH` - Path to repair configuration file
 - `--cache-dir PATH` - Directory for caching references
 - `--verbose` - Show detailed output

{linkml_reference_validator-0.4.0 → linkml_reference_validator-0.5.1}/docs/how-to/validate-titles.md

@@ -106,6 +106,23 @@ classes:
   title: "MUC1 oncoprotein blocks nuclear targeting of c-Abl"
 ```
 
+The reference may take the same shapes as for excerpts:
+
+- **A list of one** is checked like a single reference.
+- **Several references with one title field** pass if the title is any one
+  reference's title, since it names one paper. If it is none of them, a single
+  error names each reference with its own reason. A reference whose prefix is
+  skipped vouches for nothing.
+- **Reference objects with their own `title`**, listed or not, and with no title
+  field beside them, each have that title checked against their own reference.
+- **A reference with no usable identifier** (an object without `id`, an empty
+  string) is reported as an error, not skipped.
+
+With an excerpt as well, the title field is checked the same way: against the
+one reference, or, with several, passing if it is any one reference's title.
+It replaces any `title` a reference object carries. Without a title field,
+each reference object's own `title` is checked with the excerpt.
+
 ## Combined Validation
 
 When both title and excerpt fields are present, both are validated together:

{linkml_reference_validator-0.4.0 → linkml_reference_validator-0.5.1}/docs/reference/cache-format.md

@@ -117,7 +117,22 @@ assert header.model_dump()["database"] == "Orphanet"
 ```
 
 The checker derives the filename from the stored `reference_id` by replacing
-`:`, `/`, `?`, and `=` with `_` and appending `.md`. DOI filenames are compared
+`:`, `/`, `?`, and `=` with `_` and appending `.md`. A name that would exceed 200
+bytes of UTF-8 is cut on a character boundary and ends in `_` plus the first 16
+hex digits of the SHA-256 of the full `reference_id` (lowercased for a DOI), so
+a long `url:` reference fits common filesystems' 255-byte limit and two long IDs
+sharing a prefix still get different files. Shorter names are unchanged.
+
+Because the replacement is lossy, distinct references can sanitize to the same
+name, such as `url:https://ex.org/a?b=c` and `url:https://ex.org/a/b/c`. The
+first one written keeps that name. When the name is already held by a file whose
+`reference_id` is a different reference, the other one is written under the same
+name ending in `_` and the 16-hex-digit hash of its own ID instead, and the
+checker accepts that name for it. An entry is never served for a reference other
+than the one it records: a hand-written entry whose `reference_id` is a
+placeholder, or an alias spelling such as `pubmed:123` in `PMID_123.md`, is now
+treated as a miss and fetched afresh under the disambiguated name. DOI filenames
+are compared
 case-insensitively, matching the fetcher's existing behavior. Other filenames must
 match exactly. This uses the ID already stored in the file, without applying
 configured prefix aliases or requiring a currently registered reference source.

{linkml_reference_validator-0.4.0 → linkml_reference_validator-0.5.1}/docs/reference/cli.md

@@ -467,7 +467,7 @@ linkml-reference-validator repair text [OPTIONS] TEXT REFERENCE_ID
 - `--cache-dir PATH, -c PATH` - Directory for caching references
 - `--config PATH` - Path to configuration file (.yaml)
 - `--verbose, -v` - Verbose output with detailed logging
-- `--auto-fix-threshold FLOAT, -a FLOAT` - Minimum similarity for auto-fixes (default: 0.95)
+- `--auto-fix-threshold FLOAT, -a FLOAT` - Minimum similarity for auto-fixes (default: the config file's `auto_fix_threshold`, else 0.95)
 - `--help` - Show help message
 
 ### Examples
@@ -542,8 +542,8 @@ linkml-reference-validator repair data [OPTIONS] DATA_FILE
 - `--schema PATH, -s PATH` (required) - Path to LinkML schema file
 - `--target-class TEXT, -t TEXT` - Target class to validate
 - `--dry-run / --no-dry-run, -n / -N` - Show changes without applying (default: dry-run)
-- `--auto-fix-threshold FLOAT, -a FLOAT` - Minimum similarity for auto-fixes (default: 0.95)
-- `--output PATH, -o PATH` - Output file path (default: overwrite with backup)
+- `--auto-fix-threshold FLOAT, -a FLOAT` - Minimum similarity for auto-fixes (default: the config file's `auto_fix_threshold`, else 0.95)
+- `--output PATH, -o PATH` - Output file path (default: overwrite the input, keeping a `.bak` backup). When given with `--no-dry-run`, it is always written, as an unchanged copy of the input when there is nothing to fix, and the input itself is not touched, so no backup is made
 - `--config PATH` - Path to configuration file (.yaml)
 - `--cache-dir PATH, -c PATH` - Directory for caching references
 - `--verbose, -v` - Verbose output with detailed logging

{linkml_reference_validator-0.4.0 → linkml_reference_validator-0.5.1}/docs/troubleshooting.md

@@ -724,6 +724,57 @@ attributes:
     slot_uri: linkml:authoritative_reference
 ```
 
+### Citing several references for one excerpt
+
+A reference slot may be `multivalued: true`, holding a list of identifiers
+(or inlined objects with an `id`). The excerpt then passes if **any** listed
+reference contains it: a statement can rest on several papers, and its quote
+comes from one of them. References are tried in order, and the remaining ones
+are not fetched once one passes. A reference whose prefix is in
+`skip_prefixes` is not evidence either way: the other references still decide,
+and the excerpt is counted as skipped only if every listed reference is skipped.
+If none holds the excerpt, a single error names each reference with its own
+reason. The excerpt counts once in "Snippets
+checked", however many references were tried.
+
+A title slot names one paper, so it is compared only when a single reference
+is listed. With several, give each listed object its own `title`.
+
+Before this, a list was silently ignored: its excerpts were never compared and a
+fabricated quote passed with exit code 0.
+
+### Several excerpts, or evidence in a keyed mapping
+
+An excerpt slot may also be `multivalued: true`. Each listed excerpt is checked
+on its own against the reference, and reported at its own path, such as
+`supporting_text[1]`: a true excerpt beside a fabricated one does not carry it.
+A blank item, or one that is not text, is an error at its index. Before this,
+a list of excerpts crashed `validate data` with `TypeError: expected string or
+bytes-like object, got 'list'`.
+
+Evidence objects in a multivalued slot written as a mapping from identifier to
+object (`inlined_as_list: false`) are each validated, at a path such as
+`has_evidence[E1].supporting_text`; the key is used as the object's identifier
+(or `key` slot) when the object omits it. A dict is read this way only when the
+class has an identifier or key slot and none of the dict's keys is a slot of
+the class; a dict with such a key is a single object written without the list,
+and is validated as one. A compact (non-object) entry in a mapping is left to
+schema validation, and does not stop the objects beside it being checked. Before this, the objects in such a mapping were never
+visited, and a fabricated quote passed with exit code 0. The list form of the
+same slot was, and is, validated the same way.
+
+### "Reference at '...' has no usable identifier"
+
+**Cause:**
+The reference slot holds a value, but no identifier can be read from it: an
+inlined object with no `id` (or `reference_id`), an empty string, a number, or
+a `null` list item. The excerpt beside it cannot be checked, so this is an
+error rather than a silent pass.
+
+**Solution:**
+Give the reference an identifier, e.g. `reference: {id: PMID:12345678}`. An
+excerpt with no reference value at all is not affected by this check.
+
 ### "Reference base directory not found"
 
 **Cause:**
@@ -905,6 +956,20 @@ XML remains available with the existing stale-cache warning and is not rewritten
 it may still lack table rows. A later process retries the refresh. Existing
 stale HTML rejection remains unchanged.
 
+`xml_extraction_version: 2` fixes body text that version 1 could get wrong. A
+paragraph nested in another (a `<list>` or `<fig>` inside `<p>`, which JATS
+does routinely) was emitted twice; adjacent list items and lines split by
+`<break/>` were welded together ("alpha itembeta item"); HTML named entities
+such as `&nbsp;` and `&mdash;`, which PMC's external DTD defines but which is
+never loaded, were dropped and glued the words either side; and a document
+declaring its own entity in an internal DTD subset lost its whole body. A quote
+spanning any of these could fail against correct text. Version 1 XML entries
+are refreshed on the next fetch, as above. Whitespace inside a paragraph is now
+collapsed to single spaces (a no-break space included), as it already was in
+table cells.
+The HTML-entity repair applies to UTF-8 and other ASCII-compatible encodings
+only; in a UTF-16 or UTF-32 document an undeclared `&nbsp;` is still dropped.
+
 Entries that record *no* content carry their own stamp. A cache entry written
 as `content_type: unavailable` is a claim about what an extractor could not
 find, so an extractor fix can make it wrong -- and because such an entry carries

{linkml_reference_validator-0.4.0 → linkml_reference_validator-0.5.1}/src/linkml_reference_validator/cache.py

@@ -5,6 +5,7 @@ Unknown fields are preserved, so consumers can add metadata without a namespace.
 """
 
 from dataclasses import dataclass, field
+import hashlib
 from pathlib import Path
 import re
 
@@ -154,9 +155,64 @@ def legacy_cache_filename(reference_id: str) -> str:
     return _sanitized_filename(reference_id)
 
 
+#: The longest cache filename written, in UTF-8 bytes. Common filesystems allow
+#: 255; this leaves room for a ``.txt`` legacy probe and other tools' suffixes.
+MAX_CACHE_FILENAME_BYTES = 200
+
+#: Hex digits of the reference ID's SHA-256 kept in a capped filename.
+_FILENAME_HASH_CHARS = 16
+
+
 def _sanitized_filename(reference_id: str) -> str:
-    """Sanitize a reference ID into a filename, keeping everything else verbatim."""
-    return re.sub(r"[:/?=]", "_", reference_id) + ".md"
+    """Sanitize a reference ID into a filename, keeping everything else verbatim.
+
+    A name over ``MAX_CACHE_FILENAME_BYTES`` is cut on a character boundary and
+    ends in ``_`` plus a hash of the full ID, so it fits the filesystem and two
+    long IDs that share a prefix still get different files (#132). A DOI is
+    hashed case-insensitively, so case variants of one DOI still differ only in
+    case, as the fetcher's case-variant lookup and the checker expect.
+
+    >>> _sanitized_filename("PMID:1")
+    'PMID_1.md'
+    >>> name = _sanitized_filename("url:https://example.org/?q=" + "x" * 300)
+    >>> len(name.encode("utf-8")) <= MAX_CACHE_FILENAME_BYTES, name[-20:]
+    (True, '_8d32b36eb989401f.md')
+    """
+    name = re.sub(r"[:/?=]", "_", reference_id)
+    if len(name.encode("utf-8")) + len(".md") <= MAX_CACHE_FILENAME_BYTES:
+        return name + ".md"
+    return _hashed_filename(name, reference_id)
+
+
+def disambiguated_cache_filename(reference_id: str) -> str:
+    """Return the filename for a reference whose usual name another one holds.
+
+    Sanitizing maps ``:`` ``/`` ``?`` and ``=`` to ``_``, so distinct
+    references can share :func:`cache_filename`: ``url:https://ex.org/a?b=c``
+    and ``url:https://ex.org/a/b/c``. The first writer keeps the shared name;
+    the other is written here, under the same name ending in a hash of its own
+    ID, so no existing entry is renamed (#131).
+
+    >>> disambiguated_cache_filename("url:https://ex.org/a/b/c")
+    'url_https___ex.org_a_b_c_5acbea067cb82733.md'
+    """
+    key = cache_key(reference_id)
+    return _hashed_filename(re.sub(r"[:/?=]", "_", key), key)
+
+
+def _hashed_filename(name: str, reference_id: str) -> str:
+    """``name``, cut to fit, ending in ``_`` and a hash of ``reference_id``.
+
+    A DOI is hashed case-insensitively, so case variants of one DOI still
+    differ only in case.
+    """
+    digest_source = (
+        reference_id.casefold() if reference_id.upper().startswith("DOI:") else reference_id
+    )
+    digest = hashlib.sha256(digest_source.encode("utf-8")).hexdigest()[:_FILENAME_HASH_CHARS]
+    room = MAX_CACHE_FILENAME_BYTES - len(".md") - len("_") - len(digest)
+    prefix = name.encode("utf-8")[:room].decode("utf-8", errors="ignore")
+    return f"{prefix}_{digest}.md"
 
 
 def _same_reference_id(first: str, second: str) -> bool:
@@ -257,11 +313,13 @@ def _validate_cache_file(
 
     reference_id = result.frontmatter.reference_id
     expected = cache_filename(reference_id)
-    # A fragment-keyed name from before #114 is still read, so still valid.
-    accepted = {expected, legacy_cache_filename(reference_id)}
+    # A fragment-keyed name from before #114 is still read, so still valid, as
+    # is the name a reference gets when another holds its usual one (#131).
+    disambiguated = disambiguated_cache_filename(reference_id)
+    accepted = {expected, legacy_cache_filename(reference_id), disambiguated}
     matches = path.name in accepted
     if reference_id.upper().startswith("DOI:"):
-        matches = path.name.casefold() == expected.casefold()
+        matches = path.name.casefold() in {expected.casefold(), disambiguated.casefold()}
     if not matches:
         result.findings.append(
             CacheFinding(

{linkml_reference_validator-0.4.0 → linkml_reference_validator-0.5.1}/src/linkml_reference_validator/cli/lookup.py

@@ -9,7 +9,10 @@ import typer
 from ruamel.yaml import YAML
 from typing_extensions import Annotated
 
-from linkml_reference_validator.etl.reference_fetcher import ReferenceFetcher
+from linkml_reference_validator.etl.reference_fetcher import (
+    ReferenceFetcher,
+    markdown_body_lines,
+)
 from linkml_reference_validator.models import ReferenceContent
 from .shared import (
     CacheDirOption,
@@ -128,22 +131,8 @@ def _format_as_markdown(reference: ReferenceContent, fetcher: ReferenceFetcher)
     yaml_obj.dump(frontmatter, stream)
 
     lines = ["---", stream.getvalue().rstrip(), "---", ""]
-    if reference.title:
-        lines.append(f"# {reference.title}")
-        if reference.authors:
-            lines.append(f"**Authors:** {', '.join(reference.authors)}")
-        if reference.journal:
-            journal_info = reference.journal
-            if reference.year:
-                journal_info += f" ({reference.year})"
-            lines.append(f"**Journal:** {journal_info}")
-        if reference.doi:
-            lines.append(f"**DOI:** [{reference.doi}](https://doi.org/{reference.doi})")
-        lines.append("")
-        lines.append("## Content")
-        lines.append("")
-    if reference.content:
-        lines.append(reference.content)
+    # The same body the cache writer emits, so the output reads back as saved.
+    lines.extend(markdown_body_lines(reference))
     return "\n".join(lines)